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FOREWORD 


The  U.S.  Army  has  adopted  the  After  Action  Review  (AAR)  as  the  primary 
method  for  delivering  feedback  after  unit  training  exercises.  The  AAR  is  also  being 
adopted  by  proponents  within  other  Services  and  by  proponents  for  non-military  group 
training  feedback  applications.  The  U.S.  Army  Research  Institute  (ARI)  has  supported 
the  development  and  implementation  of  AAR  procedures  for  over  20  years. 

This  report  documents  the  history  of  the  AAR.  It  also  describes  the  research  and 
development  (R&D)  efforts  conducted  to  support  AAR  implementation  across  the  many 
training  situations  that  have  emerged  over  the  years  as  a  result  of  new  training 
technologies  and  force  modernization. 


EDGAR  M.  JOHNSON 
Director 


III 


FOUNDATIONS  OF  THE  AFTER  ACTION  REVIEW  PROCESS 

EXECUTIVE  SUMMARY _ _ 

Research  Requirement : 

The  after  action  review  (AAR)  is  the  Army's  method  for 
providing  performance  feedback  from  a  collective  training 
exercise.  The  U.S.  Army  Research  Institute  for  the  Behavioral 
and  Social  Sciences  (ARI)  has  performed  much  of  the  research  and 
development  (R&D)  for  the  AAR.  The  purpose  of  this  report  is  to 
review  the  evolution  of  the  AAR  process,  identify  some  of  the 
major  behavioral  science  principles  that  underlie  the  AAR,  and 
propose  directions  for  future  R&D. 

Procedure : 

The  foundations  of  AAR  were  determined  by  conducting  two 
literature  reviews.  The  first  was  a  historical  review  to 
examine  the  evolution  of  AAR  concepts.  This  review  described, 
in  chronological  order,  the  major  historical  developments  that 
have  shaped  methods  and  concepts  related  to  the  AAR  process. 

The  second  was  a  conceptual  review  of  research  on  the  AAR 
process.  This  review  was  focused  on  deriving  the  behavioral 
science  principles  that  underlie  the  AAR  process  and  was 
organized  by  general  areas  of  theory  and  research  in  behavioral 
sciences.  Finally,  the  findings  from  the  two  reviews  were 
synthesized  for  implications  for  future  research. 

Findings : 

The  history  of  the  AAR  is  relatively  short,  covering  only 
the  last  25  years.  Before  that  period,  two  influences  set  the 
stage  for  the  development  of  the  AAR: 

1.  S.  L.  A.  Marshall's  "interviews  after  combat."  This  was 
an  oral  history  technique,  first  used  in  World  War  II. 

2.  "Performance  critique."  This  was  a  feedback  technique 
used  to  provide  feedback  from  tactical  exercises  before 
the  1970s. 

The  first  AAR  methods  were  implemented  in  the  mid  1970s  in 
training  programs  developed  by  ARI  for  their  optically  based 
Tactical  Engagement  System  (TES)  training  programs.  Soon 
thereafter,  AAR  methods  were  modified  for  the  next -generation 
TES  system,  the  Multiple  Integrated  Laser  Engagement  Simulation 
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(MILES),  and  to  support  training  at  the  Army's  National  Training 
Center  (NTC) .  The  next  important  influence  on  the  AAR  process 
was  the  development  of  computer  networked  simulation  systems  in 
the  mid  1980s — in  particular,  the  Simulation  Networking  (SIMNET) 
system.  ARI  guided  the  development  of  two  generations  of  low- 
cost  systems  to  provide  SIMNET  the  capability  to  capture  and 
analyze  electronic  data  for  the  AAR. 

Since  the  initial  AAR  methods  were  developed  for  TES  and 
NTC,  ARI  has  developed  specialized  AAR  methods  for  application 
to  specific  settings,  including  staff  training  in  constructive 
simulations,  multiechelon  training  in  virtual  simulations,  and 
experimental  programs  for  training  digitized  units.  There  have 
also  been  recent  efforts  to  standardize  simulation-based 
training  programs,  which  include  the  AAR  process  (so-called 
"structured  training"  programs) .  ARI  also  has  made  input  to  an 
ongoing  program  to  standardize  and  automate  the  AAR  process: 
the  Standard  Army  After-Action  Review  System  (STAARS) .  Over 
this  short  history,  the  AAR  has  become  a  highly  regarded  process 
that  has  been  adopted  by  other  Services  and  countries . 

The  review  of  research  on  the  AAR  process  indicates  that 
AAR  principles  were  derived  from  the  information  feedback, 
performance  measurement,  cognition  and  memory,  group  processes, 
communication  theory,  and  instructional  science  areas  of 
behavioral  science.  For  each  of  these  areas,  we  identified 
specific  examples  of  AAR  methods,  practices,  or  products. 

Five  implications  for  future  research  were  developed  and 
briefly  described: 

1.  Integrate  AAR  with  other  sources  of  feedback. 

2.  Track  adherence  to  AAR  principles. 

3.  Determine  the  utility  of  AAR  process  and  products. 

4.  Determine  the  nature  of  problem-solving  in  AAR. 

5 .  Use  new  technologies  for  content  analyses . 


Utilization  of  Findings: 

The  information  provided  in  this  report  is  intended  for 
individuals  and  organizations  performing  R&D  on  AAR  processes. 
This  report  provides  background  for  research  on  the  AAR  process 
as  well  as  suggestions  for  future  research. 
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FOUNDATIONS  OF  THE  AFTER  ACTION  REVIEW  PROCESS 

Introduction 

Like  combat,  collective  training  exercises  are  complex 
events  where  the  causal  connections  between  individual 
performance,  weapons  effects,  and  mission  outcomes  are  obscured 
by  the  uncertainty,  confusion,  and  stress  of  battle  (i.e.,  "the 
fog  of  war") .  Thus,  the  answer  to  the  question,  "How  did  the 
unit  do?,"  may  not  be  immediately  obvious  to  the  exercise 
participants  or  to  those  who  control  and  observe  collective 
training  exercises.  However,  to  derive  training  value  from 
these  exercises  requires  detailed  feedback  to  the  unit  on  their 
individual  and  collective  performance  and  their  relation  to 
combat  outcomes . 

The  Army  developed  a  process,  known  as  the  after  action 
review  (AAR),  to  provide  such  feedback.  The  U.S.  Army  Research 
Institute  for  the  Behavioral  and  Social  Sciences  (ARI)  performed 
much  of  the  research  and  development  (R&D)  for  the  AAR.  The 
purpose  of  this  report  is  to  review  the  evolution  of  this 
process,  identify  some  of  the  major  behavioral  science 
principles  that  underlie  the  AAR,  and  propose  directions  for 
future  RScD. 

As  defined  in  Training  Circular  (TC)  25-20,  A  Leader' s 
Guide  to  After- Act ion  Reviews,  (U.S.  Army  Combined  Arms  Center 
[CAC] ,  1993) ,  an  AAR  "  .  .  .  is  a  professional  discussion  of  an 

event,  focused  on  performance  standards,  that  enables  soldiers 
to  discover  for  themselves  what  happened,  why  it  happened,  and 
how  to  sustain  strengths  and  improve  on  weaknesses"  (p.  1) .  In 
other  words,  the  units  perform  a  collective  self-examination  in 
which  the  more  general  question,  "How  did  the  unit  do?,"  is 
broken  down  into  three  more  specific  questions: 

1.  "What  happened  during  the  collective  training  exercise?" 
In  other  words,  AAR  participants  attempt  to  specify  the 
facts  (i.e.,  the  important  actions  and  outcomes)  of  the 
simulated  battle. 

2.  "Why  did  it  happen?"  Given  the  facts  of  the  exercise, 
the  participants  attempt  to  explain  the  causes  of 
particularly  important  actions  and  outcomes. 

3.  "How  can  units  improve  their  performance?"  Given  that 
the  previous  two  questions  are  answered,  the 
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participants  determine  appropriate  actions  to  solve 
problems  identified  in  their  performance.  Example 
actions  include  changes  to  unit  standard  operating 
procedures  (SOPs)  or  increased  training  on  basic  drills. 


These  questions  are  addressed  during  AAR  sessions,  which 
are  conducted  immediately  after  the  end  of  a  short  exercise  or 
during  logical  breaks  in  longer  exercises.  The  sessions  can  be 
formal  or  informal.  The  critical  difference  is  that  the  formal 
AARs  require  more  planning,  coordination,  and  preparation  than 
the  informal  AARs . 

AAR  sessions  are  typically  organized  by  echelon.  Platoons 
are  scheduled  first  for  about  30-45  minutes,  followed  by 
companies  for  about  1  hour,  and  then  by  battalion  and  above  for 
about  2  hours  (CAC,  1993) .  Discussion  participants  are  unit 
members  who  took  part  in  the  simulated  exercise.  The  AAR 
discussion  leader  is  typically  not  a  member  of  the  unit  being 
trained;  rather,  he  is  a  trainer  who  controlled  and  observed  the 
exercise.  Throughout  the  discussion,  the  leader  acts  as 
facilitator  of — as  opposed  to  a  participant  in — the  discussion. 
Unit  members  have  to  make  their  own  decisions  and  reach  their 
own  conclusions. 

The  leader  spends  considerable  time  preparing  for  the  AAR — 
before  the  exercise  (e.g.,  preparing  data  collection 
instruments)  and  after  the  exercise  (e.g.,  collecting  and 
aggregating  performance  data  from  various  sources) .  Typically, 
the  AAR  leader  begins  the  session  by  reviewing  the  objectives  of 
the  exercise  and  the  unit's  mission.  Then,  by  asking  open-ended 
and  leading  questions,  the  AAR  leader  gets  the  participants  to 
relate  the  significant  events  that  happened  during  the  exercise. 
For  each  event,  the  AAR  leader  tries  to  solicit  many 
perspectives  to  determine  what  really  happened.  After  the  group 
agrees  on  the  facts  of  what  happened,  the  AAR  leader  guides  the 
discussion  to  a  closer  examination  of  key  outcomes  to  determine 
why  they  happened  and  what  could  be  done  to  improve  performance. 

The  following  fictional  vignette  of  an  AAR  for  an  armor 
company  illustrates  some  possible  outcomes  from  an  AAR  session. 
An  armor  platoon  fails  to  provide  covering  fire  for  a  second 
platoon,  causing  the  second  platoon  to  be  destroyed  by  the  enemy 
when  it  is  forced  to  cross  an  open  area.  At  the  beginning  of 
the  AAR,  company  personnel  establish  that  an  implied  task  in  the 
unit's  plan  was  that  the  first  platoon  maintain  a  position  to 
observe  and  fire  on  the  enemy  should  the  second  platoon  become 
engaged.  Further  discussion  of  events  reveals  that  the  first 
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platoon  arrived  late  at  its  battle  position.  Based  on  this 
issue,  the  unit  may  decide  to  identify  different  methods  that 
can  be  used  to  ensure  that  covering  fire  is  coordinated  with 
movement  and  to  employ  one  or  more  of  these  methods  in  the 
future . 

To  facilitate  the  AAR  process,  units  can  employ  AAR  aids. 
For  example.  Figure  1  shows  a  situation  where  a  unit  is 
observing  a  replay  of  a  portion  of  an  exercise  as  it  discusses 
mission  execution.  It  is  important  to  remember  that  the  focus 
of  the  AAR  is  on  unit  discussions  of  exercise  events,  and  AAR 
aids  are  one  of  many  tools  used  to  guide  and  encourage 
participation.  Figure  2  shows  a  case  where  the  attention  of 
unit  members  is  focused  on  the  comments  being  made  by  one  of 
their  own. 


Figure  1.  A  unit  observes  a  replay  of  an  exercise 
segment  at  an  AAR  station  from  an  "out -the -window"  view 
as  members  discuss  what  happened,  why  it  happened,  and 
how  to  improve  or  sustain  performance. 

In  addition  to  a  detailed  analysis  of  combat  outcomes  and 
unit  performance,  the  AAR  session  often  includes  the 
identification  of  areas  that  deserve  special  attention  for 
training.  The  AAR  leader  may  identify  the  critical  tasks 


3 


Figure  2.  Participation  of  unit  members  in  discussions 

is  key  to  the  AAR  process. 

performed  during  the  exercise  and,  for  each,  ask  the 
participants  whether  they  think  their  unit  is  proficient  (and 
should  sustain  training  practices)  or  deficient  (and  should 
improve  training) .  The  commander,  in  his  role  as  the  unit's 
primary  trainer,  uses  all  the  feedback  from  the  AAR  to  assess 
his  unit's  performance. 

The  next  two  sections  this  report  detail  the  foundations  of 
the  AAR.  These  foundations  were  determined  by  conducting  two 
reviews  of  the  literature.  The  first  was  a  historical  review  to 
determine  the  evolution  of  AAR  concepts.  The  second  was  a 
conceptual  review  of  recent  research  on  the  AAR  process.  This 
second  review  focused  on  deriving  the  behavioral  science 
principles  that  underlie  the  AAR  process  and  was  organized  by 
general  areas  of  theory  and  research  in  behavioral  sciences . 

The  final  section  synthesizes  implications  from  the  reviews  into 
implications  for  future  research  opportunities. 


Historical  Foundations 

The  concept  of  the  AAR  has  evolved  over  the  last  25  years — 
a  period  that  has  seen  dramatic  improvements  in  collective 
training  technology.  At  the  beginning  of  this  period,  Army 
units  conducted  relatively  free-play,  force-on-force  exercises 
that  were  evaluated  in  field  settings  by  human  umpires  who  were 
ill-equipped  and  poorly  positioned  to  observe  performance. 
Performance  feedback,  therefore,  was  based  on  the  umpires' 
subjective  estimates  of  mission  outcomes  (e.g.,  "hits"  and 
"kills"  of  soldiers  and  systems)  and  judgments  of  overall 
mission  success . 

At  present.  Army  units  train  in  live,  virtual,  or 
constructive  simulation  environments^  that  allow  automated 
measurement  of  a  wide  array  of  performance  indicators,  including 
tactical  movement,  engagement  outcomes,  and  communications. 
Moreover,  unit  performance  is  often  assessed  in  the  context  of  a 
highly  structured  exercise  based  on  a  standardized  training 
support  package  that  specifies  training  objectives,  training 
events,  and  performance  measures.  This  marked  progress  in 
collective  training  and  evaluation  methods  parallels  the 
development  of  methods  and  technologies  related  to  the  AAR 
process.  This  section  discusses  some  of  the  major  historical 
developments  that  have  shaped  methods  and  concepts  related  to 
the  AAR  process . 

Antecedents  of  the  After  Action  Review  (AAR) 

An  important  historical  antecedent  of  the  AAR  is  the  combat 
.interview  developed  by  military  historian  Samuel  Lyman  Atwood 
(S.  L.  A.)  Marshall  (1900-1977)  (Gubler,  1997;  Bosley,  Onoszko, 
Knerr,  &  Sulzen,  1979;  Word,  1987) .  S.  L.  A.  Marshall  was  a 
member  of  a  team  of  Army  historians  tasked  to  document  the 
events  of  World  War  II  as  they  unfolded  (Everett,  1992) . 
Initially  assigned  to  the  Pacific  theater,  Marshall  was 
frustrated  in  his  attempts  to  reconstruct  a  battle's  sequence  of 
events  from  normal  historical  data  sources,  such  as  official 
records  and  documents.  Drawing  on  his  experience  as  a 


1  These  three  types  of  simulation  vary  primarily  with  respect 
to  realism.  Constructive  simulations  rely  on  simulated 
operators,  equipment,  and  situations;  virtual  simulations  rely 
on  real  operators  in  simulated  equipment  and  simulated 
situations;  and  live  simulation  rely  on  real  operators  and  real 
equipment  in  simulated  situations  (Sikora  &  Coose,  1995) . 
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journalist  and  as  a  professional  historian,  Marshall  developed 
an  oral  history  method,  which  he  called  the  "interview  after 
combat,"  where  he  assembled  battle  participants  immediately 
after  the  fighting  had  ended  and  conducted  group  interviews. 
Marshall  and  his  associates  would  ask  questions  designed  to  lead 
the  participants  through  the  battles.  These  interviews  provided 
a  unique  source  of  detailed  information  about  what  occurred 
during  the  battle.  Their  efforts  expanded,  and,  by  the  end  of 
the  war,  over  2,000  such  interviews  had  been  conducted  in  the 
European  theater  alone  (Everett,  1992) .  S.  L.  A.  Marshall  and 
his  colleagues  extended  their  work  to  the  Pacific  theater  and, 
later,  to  the  Korean  and  Vietnam  wars.  The  combat  interview 
remains  a  primary  method  of  military  historical  research.  It 
became  a  model  for  the  AAR  by  providing  a  method  for  eliciting 
objective  and  reliable  information  on  combat  performance. 

Another  influence  on  development  of  the  AAR,  albeit  a 
negative  one,  was  the  performance  critique.  This  was  the 
dominant  method  for  providing  performance  feedback  before  the 
development  of  the  AAR.  Before  the  time  when  methods  were 
^V9^il3.ble  to  assess  battle  damage  objectively,  the  outcomes  of 
simulated  battles  were  determined  subjectively  by  human  umpires. 
For  instance,  umpires  determined  the  outcome  of  an  assault  on  a 
defended  position  by  subjectively  combining  factors  such  as 
numerical  superiority,  swiftness,  and  noise  level  of  the 
attacking  unit  (Gubler,  1997) .  Well  after  the  battle  had  ended, 
the  senior  umpire  would  determine  an  estimated  outcome  along 
with  his  interpretation  of  tactical  performance,  which  would  be 
provided  in  the  performance  critique.  This  feedback  often 
lacked  credibility  because  it  was  based  largely  on  subjective 
opinion  rather  than  on  objectively  measured  performance  and 
battle  outcomes.  The  lecture- formatted  critique  was  often 
negative  in  tone,  thereby  fomenting  resentment  among 
participants  and  resistance  to  the  umpire's  criticism. 

By  the  early  1970s,  a  rapidly  emerging  consensus  held  that 
the  traditional  performance  critique  was  not  an  effective 
performance  feedback  method  and  was  actually  counterproductive 
to  the  goal  of  enhancing  unit  performance.  Those  who 
implemented  and  participated  in  collective  training  realized 
that  some  form  of  feedback  was  needed.  This  feedback 
requirement  fueled  the  development  of  the  tactical  engagement 
simulation  (TES) ,  which  was  designed  to  provide  more  accurate 
and  objective  feedback.  Notably,  in  the  early  development 
t^isls  of  TES,  participants  would  spontaneously  meet  to  discover 
and  discuss  TES  feedback  to  determine  what  happened  during 
simulated  battles  (Shriver,  cited  in  Sulzen,  1986) . 
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To  realize  the  benefits  of  new  simulation  techniques,  the 
Army  determined  that  a  new  approach  for  providing  performance 
feedback  was  needed.  ARI  researchers  led  the  effort  to  develop 
and  implement  this  new  method,  which  became  the  AAR.  The  AAR 
was  designed  to  incorporate  the  objectivity  and  non-punitive 
atmosphere  that  was  characteristic  of  S.  L.  A.  Marshall's 
interviews  after  combat  (Bosley,  Osnoszko,  Knerr,  &  Sulzen, 

1979) .  At  the  same  time,  the  AAR  design  avoided  some  of  the 
more  egregious  deficiencies  of  the  performance  critique. 

Table  1  compares  some  of  the  features  of  this  new  approach 
with  corresponding  features  of  the  traditional  performance 
critique.  As  can  be  seen,  the  new  AAR  approach  for  providing 
feedback  differed  from  the  old  performance  critique  in  both  form 
and  content.  The  final  entry, in  Table  1  identifies  a  critical 
difference:  whereas  the  performance  critique  is  based  on 
subjective  judgment,  the  AAR  is  based  on  objective  performance 
indicators.  The  technologies  that  enabled  more  objective 
methods  for  obtaining  performance  data  are  discussed  in  the  next 
section. 

Development  of  Simulation-Based  Training  Systems 

As  implied  in  the  definition  of  the  AAR,  an  initial  and 
fundamental  requirement  is  to  determine  what  happened  during  a 
simulated  battle.  Thus,  a  significant  impetus  to  the  evolution 
of  systematic  AAR  procedures  was  the  development  of  objective 
methods  for  credible  casualty  and  battle  damage  assessments 
(BDAs) .  The  following  sections  describe  modern  simulation 
methods  that  were  developed  in  the  last  25-30  years  to  provide 
this  capability  and  that  contributed  to  the  development  of 
systematic  AAR  processes. 

Tactical  Engagement  Simulation  (TES) 

In  1971,  General  W.  C.  Westmoreland,  then  Chief  of  Staff  of 
the  Army,  directed  a  board  of  officers  to  consider  new 
collective  training  approaches  that  were  motivating  and 
effective  (Gorman,  1992) .  The  board  initially  investigated  a 
system  of  laser-based  emitters  and  detectors  for  simulating 
infantry  and  armor  close  weapons  effects.  This  system  was  later 
named  the  Multiple  Integrated  Laser  Engagement  System  (MILES) . 
Because  the  MILES  technology  would  require  at  least  12  years  to 
develop  and  field,  the  board  also  tasked  ARI  to  develop  rapidly 
an  alternative,  optically  based  TES  system  to  help  bridge  the 
gap  (Gorman,  1992) . 
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Table  1 

Contrast  of  Performance  Critiques  and  AARs 


Characteristics 

Performance 

Feedback  Method 

of  Feedback 
Sessions 

Traditional  Performance 
Critique 

AAR 

Soldier 

participation 

Soldiers  are  passive 
members  of  an  audience. 

Soldiers  are  active 
participants  in  a 
discussion. 

Main  topic  of 
discussion 

Errors  committed. 

Sequence  of  events. 

Direction  of 
communication 

One-way  (from  leader  to 
participants) . 

Two-way. 

Atmosphere 

Defensive. 

Open  to  suggestion. 

Instructional 

style 

Traditional  lecture . 

Guided  discovery  learning. 

Source  of 
information: 
why  it  happened? 

Exercise  leader  and 
controllers . 

Participants  and  members 
of  the  opposing  force 
(OPFOR)  and  exercise 
leaders  and  controllers. 

Source  of 
Information: 
what  happened? 

Sub j  ec t i ve  j  udgment . 

Objective  performance 
indicators. 

ARI's  work  on  an  optically  based  TES  system  began  in  1972, 
and  the  first  product  was  fielded  in  1973.  This  initial  effort 
produced  a  method  for  training  small,  dismounted  infantry  units 
called  Squad  Combat  Operations  Exercises  (Simulation)  or  SCOPES. 
The  basic  hardware  for  SCOPES  was  a  6X  telescope  mounted  on 
every  participant's  rifle  and  helmets  with  numbers  that  uniquely 
identified  each  soldier  in  the  exercise.  To  engage  the 
opposition,  a  participant  would  identify  an  opponent's  number, 
fire  a  blank  round,  and  report  the  number  to  a  controller.  The 
controller  would  report  to  a  counterpart  controller  of  the 
opposition  forces,  who  would  then  assess  the  participant  whose 
number  was  identified  as  a  casualty.  The  SCOPES  approach  was 
extended  to  combined  arms,  armor  and  antiarmor  units  in  a 
product  called  REALTRAIN  (Realistic  Training) . 

SCOPES  and  REALTRAIN  were  significant  because  they  were  the 
first  TES  technologies  to  provide  trainers  and  participants  an 
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objective  representation  of  "ground  truth"  from  tactical 
engagement  outcomes.  As  such,  SCOPES  and  REALTRAIN  provided  an 
appropriate  context  for  the  first  reported  implementations  of 
the  AAR  process  (Bosley,  Onoszko,  Knerr,  &  Sulzen,  1979;  Bosley, 
Onoszko,  &  Sevilla,  1979) .  The  training  programs  that 
incorporated  these  TES  technologies  and  the  AAR  methods  were 
evaluated  in  a  series  of  ARI  research  projects  (e.g..  Banks, 
Hardy,  Scott,  Kress,  &  Word,  1977;  Root,  Epstein,  et  al . ,  1976; 
Root,  Hayes,  Word,  Shriver,  &  Griffin,  1979;  Scott,  Banks, 

Hardy,  &  Sulzen,  1979) .  These  studies  confirmed  that  the  SCOPES 
and  REALTRAIN  training  programs  were  effective  and 
motivationally  engaging  methods  for  training  tactical  skills; 
however,  the  AAR  components  of  the  programs  were  not  evaluated 
separately.  The  demonstrable  successes  of  these  training 
packages  led  to  their  rapid  adoption  by  the  U.S.  Army's  Training 
and  Doctrine  Command  (TRADOC) .  ARI  helped  to  write  the  TCs 
published  by  the  Infantry  School  for  SCOPES  in  1973  and  by  the 
Armor  School  for  REALTRAIN  in  1975. 

MILES  was  introduced  in  1980,  but  it  was  not  put  into 
general  use  until  mid-decade  (Gorman,  1992) .  Once  fielded, 
however,  its  impact  was  immediate  and  significant.  Zeidner  and 
Drucker  (1988)  characterized  MILES  as  "  .  .  .  one  of  the  most 

revolutionary  changes  in  combat  training  ever  introduced" 

(p.  186) .  Compared  with  SCOPES  and  REALTRAIN,  the  laser-based 
MILES  system  significantly  increased  the  accuracy  of  determining 
hits  and  misses.  Moreover,  MILES  greatly  reduced  the  number  of 
controllers  required  to  support  a  TES  exercise  and  provided  more 
immediate  feedback  to  participants.  Although  MILES  equipment 
was  fielded  to  units  on  an  Army-wide  basis,  its  initial 
implementation  was  at  the  National  Training  Center  (NTC) ,  where 
force -on -force  maneuvers  were  first  conducted  in  1982. 

As  the  NTC  rapidly  became  the  Army' s  premier  combat 
training  setting,  ARI  adapted  the  AAR  process  first  developed 
for  SCOPES  and  REALTRAIN  to  the  innovations  at  NTC,  which 
offered  several  new  resources  and  technologies  for  collecting 
and  documenting  objective  performance  data.  Most  notably,  MILES 
extended  ARI's  earlier  TES  methods  for  providing  objective 
performance  data.  In  addition  to  MILES,  other  components  of  the 
NTC  had  a  significant  impact  on  the  AAR  process,  including  its 
sophisticated  instrumentation  system  for  exercise  control  and 
data  collection,  a  highly  trained  OPFOR,  and  expert 
observer/controllers  (0/Cs) .  The  NTC's  reliance  on  the  AAR  as 
their  primary  feedback  method  increased  the  prominence  of  the 
AAR  as  an  effective  training  method.  Accordingly,  ARI  has 
continued  to  adapt  the  NTC  methods  to  other  training  contexts. 
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including  home  station  field  exercises  (Scott,  1983;  1984)  and 
the  Army  Training  Battle  Simulation  System  (ARTBASS) (Kaplan  & 
Fallesen,  1986) . 

Computer  Networked  Simulation 

The  development  of  the  Simulation  Networking  (SIMNET) 
system  also  provided  a  great  stimulus  to  the  development  and 
refinement  of  AAR  processes.  Developed  and  fielded  in  the 
1980s,  SIMNET  was  the  first  networked  system  of  virtual 
simulators  to  be  used  for  collective  training  of  heavy  combat 
units  (Alluisi,  1991) .  Compared  with  TES  and  other  live 
simulations,  SIMNET  has  greatly  increased  the  amount  of 
information  that  can  be  captured  and  used  in  the  AAR.  As 
summarized  by  Meliza  (1996),  "...  we  went  from  a  situation 
where  vehicle  status  data  were  limited  to  periodic  updates 
regarding  the  location  and  damage  status  of  vehicles  to  the 
point  of  having  near  continuous  updates  on  the  locations  of 
vehicles,  the  speed  at  which  vehicles  were  moving,  the 
orientation  of  vehicles,  engine  speeds,  the  orientation  of  gun 
tubes,  the  elevation  of  gun  tubes,  fuel  levels,  and  ammunition 
levels  .  .  . "  (p .  13 ) . 

By  providing  the  ability  to  reinitialize  scenarios  rapidly, 
networked  simulations  also  afford  units  repeated  opportunities 
to  correct  problems  immediately  and  apply  solutions  identified 
in  previous  AARs .  At  the  same  time,  networked  simulations 
shorten  the  acceptable  delay  in  providing  the  feedback  from 
tactical  exercises  (Meliza  &  Brown,  1996) .  In  live  simulations 
(e.g.,  those  employed  at  NTC) ,  units  typically  perform  logistic 
and  maintenance  functions,  such  as  vehicle  maintenance  and 
resupply,  immediately  after  an  exercise.  In  SIMNET,  these 
activities  are  either  not  performed  or  are  foreshortened. 
Therefore,  units  have  come  to  expect  their  AAR’  soon  after  an 
exercise  ends. 

Early  Simulation  Network  (SIMNET)  Feedback  Aids  and 
Limitations .  Despite  the  inherent  potential  of  SIMNET  to 
provide  detailed  performance  data,  the  initial  training  version 
of  SIMNET  provided  no  features  to  support  automated  performance 
measurement . 2  in  fact,  the  initial  concept  for  SIMNET  called  for 


2  This  section  refers  to  the  initial  training  version  of 
SIMNET,  called  SIMNET-T,  as  first  implemented  at  Fort  Knox, 
Kentucky.  Later  SIMNET-T  sites,  particularly  those  designed  to 
support  gunnery  competition,  included  more  extensive  automated 
performance  measurement  systems,  as  did  developmental  sites. 
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using  battle  outcomes  as  the  primary  performance  indicator.  The 
developers  of  SIMNET  proscribed  against  the  use  of  instructors, 
controllers,  or  umpires.  The  AARs  were  to  be  performed  solely 
by  the  chain  of  command  as  they  would  in  actual  combat :  "Just 
as  in  combat,  there  are  no  overlords  in  this  type  of  exercise 
other  than  the  chain  of  command"  (Thorpe,  1987) .  Nevertheless, 
two  ad  hoc  capabilities  to  document  and  observe  performance  were 
added  soon  after  the  system  was  developed: 

1.  The  Plan-View  display,  which  provided  a  top-down 
("bird's  eye")  view  of  the  battlefield. 

2 .  The  Stealth-View  display,  which  provided  a  virtual 
( "out -the -window" )  view  of  the  battlefield  from  a 
vehicle  that  was  not  visible  to  participants  (hence, 
"stealth") . 

When  coupled  with  a  data  logger  that  recorded  network  data 
stream  elements  (or  Protocol  Data  Units  [PDUs] )  associated  with 
a  simulated  exercise,  both  the  Plan-View  and  Stealth-View 
displays  provided  the  capability  to  replay  all  or  parts  of  the 
battle  at  variable  speed  and  perspective.  Although  these 
features  potentially  provided  a  wealth  of  information  for  AARs, 
the  displays  were  initially  used  primarily  to  prepare  and 
control  ongoing  exercises.  In  addition,  Goldberg  and  Meliza 
(1993)  noted  several  specific  deficiencies  in  the  design  of  the 
initial  SIMNET-T  training  facilities  with  regard  to  the  AAR 
process : 

•  The  SIMNET  training  facility  allowed  many  exercises  to  be 
conducted  concurrently  but  supported  only  one  AAR  at  a 
time.  Further,  the  system  could  not  be  used  for  an  AAR 
if  it  was  employed  to  support  exercise  control  functions 
for  an  ongoing  exercise. 

•  The  system  did  not  provide  data  summary  tables  and 
graphics.  Despite  the  fact  that  SIMNET  could  provide  the 
most  extensive  collection  of  objective  data  ever 
available  to  examine  unit  perf oirmance ,  it  did  not  include 
tools  for  analyzing  these  data  in  a  training  environment. 

•  Replaying  an  exercise's  history  was  a  slow  and  cumbersome 
way  to  review  unit  performance.  For  example,  one  could 
watch  an  engagement  lasting  15  minutes  to  find  out  that 


called  SIMNET-D,  at  Fort  Knox,  Kentucky,  and  Fort  Rucker, 
Alabama . 
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only  two  of  the  four  tanks  in  a  platoon  fired  their 
weapons,  whereas  one  could  look  at  a  table  showing  rounds 
fired  by  individual  tanks  and  immediately  know  who  fired 
and  who  did  not . 

Automated  Aids  Appended  to  the  Simulation  Network  (SIMNET) . 
To  improve  the  automated  performance  measurement  capabilities  of 
SIMNET,  ARI  guided  the  development  of  two  generations  of 
automated  aids  for  the  training  system:  the  Unit  Performance 
Assessment  System  (UPAS)  and  the  Automated  Training  Analysis  and 
Feedback  System  (ATAFS) . 

In  1990,  ARI  initiated  the  development  of  the  first 
generation  of  automated  performance  measurement  systems  for 
SIMNET:  the  UPAS,  which  was  an  inexpensive,  DOS-based  computer 
system  that  collected  and  analyzed  information  from  the  SIMNET 
data  stream.  The  purpose  of  UPAS  was  to  provide  AAR  leaders  the 
capability  to  create  and  experiment  with  new  aids  and  displays, 
which  could  be  produced  without  special  knowledge  of  computer 
programming.  According  to  White,  McMeel,  and  Gross  (1990),  the 
prototype  UPAS  provided  for  two  types  of  AAR  aids : 

•  A  top-down  Plan-View  display  that  provided  a  bird's-eye 
view  of  the  battlefield. 

•  Graphs  and  tables  that  summarized  the  results  of  the 
battle. 

The  UPAS  also  provided  graph  and  table  editors  that  non¬ 
programmers  could  use  to  modify  existing  graph  and  table 
displays  and/or  to  create  new  ones.  Unfortunately,  this  useful 
feature  of  UPAS  was  not  incorporated  into  successor  systems. 

The  variety  of  AAR  aids  that  could  be  produced  with  the 
UPAS  increased  with  the  continued  development  of  this  system, 
and  tools  were  added  to  help  users  manage  the  presentation  of 
AARs  (Meliza  and  Tan,  1996) .  Many  of  the  changes  made  in  the 
UPAS  were  in  response  to  the  results  of  user  feedback  (Shlecter, 
Bessemer,  Wade,  and  Nesselroade,  1994) .  UPAS  aids  were  designed 
to  capture  measures  of  unit  performance  that  were  difficult  to 
collect  using  SIMNET  exercise  replays.  An  example  aid  is  the 
"Exercise  Timeline,"  which  displays  a  variety  of  battle  events 
on  a  timeline.  Such  displays  might  reveal  when  a  unit  crosses  a 
key  battlefield  location,  such  as  phase  lines.  These  data  might 
be  used  to  show  that  a  unit  made  many  long  halts  during  its 
movement.  This  information  could  then  be  used  to  explain  why 
the  unit  failed  to  arrive  at  its  attack  position  in  time. 
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UPAS  offered  advantages  over  those  provided  by  AAR  aids 
built  into  SIMNET.  Although  UPAS  was  improved  continually 
throughout  its  development,  Meliza  and  Brown  (1996)  pointed  out 
some  serious  shortcomings  that  could  not  be  overcome  through 
software  modification. 

The  fundamental  shortcoming  of  UPAS  was  that  it  was 
implemented  on  an  early-generation,  DOS-based  machine  that  could 
only  execute  one  task  at  a  time.  Three  specific  problems  were 
identified: 

1.  During  exercises,  UPAS  was  fully  occupied  collecting 
exercise  data;  therefore,  the  task  of  selecting  and 
creating  AAR  aids  could  not  begin  until  the  end  of  the 
exercise  (ENDEX) .  As  a  result,  the  preparation  of  UPAS 
feedback  regularly  exceeded  the  10-20  minute  time  limit 
that  was  generally  accepted  as  the  minimal  acceptable 
delay — despite  software  improvements  designed  to  speed 
up  the  system. 

2.  UPAS  could  not  simultaneously  collect  voice  data  linked 
to  events  while  collecting  network  data.  As  a  result, 
it  could  not  record  and  play  back  radio  communications 
data  synchronized  with  visual  data.  The  lack  of  voice 
data  was  deemed  a  serious  shortcoming  in  determining 
what  happened  during  an  exercise. 

3.  UPAS  did  not  provide  trainers  help  in  deciding  what  AAR 
aids  would  be  useful  for  a  particular  exercise  and  how 
they  might  be  used.  This  feature  was  crucial  in 
determining  whether  the  system  would  be  used  as 
designed. 

With  the  limitations  of  UPAS  in  mind,  the  Army's 
Simulation,  Training,  and  Instrumentation  Command  (STRICOM)  and 
the  Defense  Advanced  Research  Projects  Agency  (DARPA)  funded  and 
ARI  monitored  the  development  of  the  next  generation  of 
automated  AAR  aids:  ATAFS .  The  ATAFS  project's  goal  was  to 
demonstrate  the  capability  to  provide  AAR  leaders  a  menu  of 
automatically  and  manually  generated  aids  for  use  in  the  AAR 
within  10  minutes  after  the  end  of  a  simulation  exercise  (Brown, 
Wilkinson,  et  al . ,  1996).  The  ATAFS  was  able  to  meet  this  goal 
by  employing  platforms  with  multi-tasking  capabilities  and  by 
using  a  knowledge  database  to  automatically  generate  AAR  aids 
during  exercises.  As  a  result,  the  ATAFS  was  able  to  provide  a 
bin  of  candidate  AAR  aids  available  for  use  at  ENDEX. 
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Development  of  Specialized  After  Action  Review  (AAR)  Methods 

The  unique  needs  of  particular  training  environments  have 
demanded  the  development  of  specialized  AAR  methods.  As 
summarized  in  Table  2,  ARI  has  developed  specialized  methods  and 
S^idance  for  delivering  AARs  in  a  variety  of  specific  training 
and  simulation  environments.  In  Table  2,  the  descriptions  of 
the  training  environments  are  suggestive  of  the  situational 
variables  (echelon,  type  of  simulation,  and  so  forth)  that 
necessitated  the  development  of  specialized  guidance. 

Bosley,  Onoszko,  Knerr,  and  Sulzen  (1979)  developed  the 
first  ARI  document  to  guide  Army  leaders  in  conducting  AARs.  It 
was  based  upon  observations  of  AARs  in  association  with  the  use 
of  REALTRAIN .  Scott  and  Fobes  (1982)  provided  a  guidebook  for 
conducting  AARs  at  the  Army's  NTC.  This  guidebook  was  based,  in 
part,  on  observations  of  AARs  in  association  with  the  use  of 
REALTRAIN' s  replacement,  the  MILES  system.  It  was  modified 
twice  to  fit  new  audiences.  As  part  of  ARI's  efforts  to  support 
the  fielding  of  MILES  within  units  across  the  Army,  Scott  (1983) 
modified  the  guidebook  for  use  by  trainers  at  home  station. 
Kaplan  and  Fallesen  (1986)  tailored  the  guidebook  to  fit  AARs 
for  the  ARTBASS  constructive  training  simulation. 

In  a  1986  ARI  report,  L.  E.  Word  provided  a  new  source  of 
information  for  conducting  AARs,  based  on  his  tour  of  duty  as  a 
senior  0/C  at  the  Army's  NTC.  Among  other  things.  Word  stressed 
the  importance  of  having  the  OPFOR  members  participate  in  the 
AAR.  These  OPFOR  participants  bring  two  sources  of  information 
designed  to  improve  the  perspective  of  the  members  of  the  unit 
being  trained: 

1.  An  additional  viewpoint  on  exercise  events,  as  viewed 
from  the  perspective  of  the  enemy. 

2.  Casualty  exchange  data  (casualties  inflicted  and 
sustained)  that  would  not  normally  be  known  to  units 
because  of  the  "fog  of  war." 

Certain  strategies  presented  in  Word's  (1987)  report  for 
improving  the  quality  of  AARs  are  applicable  across  training 
environments,  but  others  have  to  be  modified  to  fit  different 
environments.  This  observation  mirrors  the  trend  demonstrated 
by  ARI  in  1982  and  beyond  to  tailor  AAR  guidance  for  specific 
situations.  Since  1982,  the  guidance  developed  by  ARI  for 
conducting  AARs  tends  to  address  specific  training  situations. 
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such  as  those  described  in  Table  2.  The  column  that  provides 


Table  2 

AAR  Guidance  for  Specific  Training  Environments 


Program  Name 

Simulation  Environment 

ARI  Publication (s) 

Realistic  Training 
(REALTRAIN) 

Live,  optically  based  simulation 
for  platoon-  and  company- level 
training 

Bosley,  Onoszko, 

Knerr,  Sc  Sulzen 
(1979) 

National  Training 
Center  (NTC) 

Live,  laser-based  simulation  and 
instrumented  range  for 
battalion-  and  brigade -level 
training 

Scott  Sc  Fobes 
(1982);  Meliza, 

Sulzen,  Atwood,  & 
Zimmerman  (1987) 

Army  Training  and 
Evaluation  Plan 
(ARTEP) 

Live,  laser-based  simulation  for 
squad-,  platoon-,  and  company- 
level  training 

Scott  (1984) 

Army  Training  Battle 
Simulation  System 
(ARTBASS) 

Constructive  computer-based 
simulation  for  battalion  staffs 

Kaplan  &  Fallesen 
(1986) 

Precision  Range  and 
Integrated  Maneuver 
Exercise  (PRIME) 

Live,  laser-based  simulation  and 
instrumented  range  for  crew-  and 
platoon- level  tank  training 

Witmer  (1990) 

Multi -Service 
Distributed  Training 
Testbed  (MDT2) 

Live,  virtual,  and  constructive 
simulation  for  training  multi - 
service  personnel  in  close  air 
support 

Bell  et  al.  (1997) ; 
Mirabella,  Sticha,  & 
Morrison  (1997) 

Virtual  Training 
Program  (VTP) 

Structured  programs  of  virtual 
and  constructive  simulation 
exercises  from  crew  through 
brigades 

Roger  et  al .  (1996) 

Force  XXI  Training 
Program  (Combined 
Arms  Operations  at 
Brigade  Level 
Realistically 
Achieved  Through 
Simulation  [COBRAS] ) 

Structured  programs  of  virtual 
and  constructive  simulation  for 
brigade  and  battalion  staffs 

R.  C.  Campbell  et 
al.  (1999) 

Staff  Group  Trainer 
(SGT) 

Constructive  simulation  training 
programs  for  battalion  and 
brigade  staff  groups 

Roger  etal.  (1998) 

Simulation-Based 

Multiechelon 

Training  Program  for 
Armor  Units -Digital 
(SIMUTA-D) 

Virtual  and  constructive 
simulation  for  battalion  staffs 
of  digitized  forces 

Winsch,  Garth, 

Ainslie,  & 

Castleberry  (1996) 

15 


descriptions  of  training  environments  suggests  that  AAR  methods 
may  differ  as  a  function  of  the  simulation  (live,  virtual,  or 
constructive)  and  echelon  and  whether  the  training  is  structured 
or  not  structured.  As  the  options  for  training  environments 
increase,  the  need  for  tailored  guidance  will  also  expand. 

Various  combinations  of  training  environments  and  training 
audiences  appear  to  present  certain  unique  challenges  to  the  AAR 
process;  however,  at  the  same  time,  many  combinations  also  offer 
certain  unique  benefits.  For  example,  consider  Word's  strategy 
of  having  OPFOR  take  part  in  an  AAR  to  provide  units  with 
information  about  how  the  enemy  perceived  their  actions.  In 
most  virtual  exercises,  a  computer-generated  force  represents 
the  enemy,  so  no  human  OPFOR  take  part  in  the  AAR.  Similarly, 
feedback  resources  for  live  training  may  be  limited  when  a 
series  of  AARs  are  conducted,  beginning,  for  example,  at  the 
platoon,  company,  and  then  battalion  level  after  a  battalion 
exercise.  In  the  latter  situation,  bringing  in  that  specific 
part  of  the  OPFOR  with  which  a  particular  platoon  or  company 
interacted  would  be  difficult,  so  platoons  and  companies  have  to 
look  elsewhere  for  the  information  they  need. 

ARI  continues  to  analyze  the  feedback  and  AAR  requirements 
of  new  developments  in  training  and  force  modernization  to 
determine  whether  new  methods  or  aids  are.  required  (Brown, 
Nordyke,  et  al . ,  1998).  ARI  has  focused  recent  efforts  on 
developments  that  have  immediate  impact  on  feedback 
requirements:  the  new  information  sources  provided  by  digitized 
command,  control,  communications,  computers,  and  intelligence 
(C*!)  systems.  Before  the  advent  of  digital  communications,  an 
0/C  monitored  voice  communications  and  exercises  while  playing 
the  roles  of  higher  headquarters,  supporting  units,  and  adjacent 
\inits.  With  digital  C^I  systems,  however,  much  of  the 
communication  of  tactical  information  is  displayed  visually  on 
computers  located  in  combat  vehicles. 

Monitoring  all  sources  of  verbal  and  visual  information  is 
difficult  if  not  impossible.  The  increased  emphasis  on 
monitoring  communications  makes  it  more  difficult  for  an  0/C  to 
perform  other  duties,  such  as  role  playing,  exercise  control, 
and  so  forth.  Moreover,  without  the  input  from  all  digital 
communications,  the  ability  of  the  0/C  to  prepare  AARs  on  the  C"*! 
aspects  of  the  mission  has  been  reduced.  To  aid  the  0/C  in 
developing  AARs  for  digitized  units,  ARI  is  currently  monitoring 
the  STRICOM- funded  development  of  the  C^I  Training  Analysis  and 
Feedback  System  (CTAFS)  for  implementation  in  the  Close  Combat 
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Tactical  Trainer  (CCTT) ,  the  system  that  will  eventually  replace 
SIMNET  (Brown,  Metzler,  Riede,  &  Wonsewitz,  1996;  Nordyke, 
Gerlock,  Montague,  Huysoon,  &  Bucher,  1998) .  CTAFS  is  designed 
to  provide  AAR  aids  and  manage  the  AAR  presentation  for  digital 
units  in  a  manner  analogous  to  ATAFS,  which  assists  the 
development  of  AARs  for  conventional  units . 

Standardization  of  Collective  Training 

The  so-called  "combat  model"  that  dominated  thinking  in  the 
early  days  of  SIMNET  development  implied  that  training  would  be 
conceived  and  conducted  by  units  without  benefit  of  observers, 
controllers,  or  instructors.  This  model  presented  fundamental 
problems  for  training.  For  one,  the  concept  of  the  unit's 
nearly  absolute  autonomy  in  training  matters  ran  counter  to  the 
widely  held  principle  that  Army  training  should  be  standardized 
across  the  force.  Perhaps  more  fundamentally,  this  laissez 
faire  concept  of  simulation  training  had  the  potential  to 
undermine  SIMNET  and  other  simulations'  potential  for  improving 
performance  on  tactical  tasks.  Partly  as  a  reaction  to  these 
specific  problems,  the  Army  started  to  standardize  the  methods 
used  to  conduct  simulation-based  tactical  training. 

Structured  Training 

In  the  early  1990s,  ARI  developed  a  standardized  approach 
to  simulation-based  tactical  training  called  "structured 
training."  In  general,  structured  training  has  been  defined  as 
a  coherent  program  of  instruction  and  practical  exercises  for 
learning  and  mastering  specific  learning  objectives  (ARI,  1996) . 
In  the  context  of  simulation-based  tactical  training,  Campbell, 
Deter,  and  Quinkert  (1997,  p.  6)  identified  the  following  key 
features  of  structured  training : 

•  A  focus  on  the  performance  of  selected  critical  tasks. 

•  Standardized  exercise  control  to  ensure  practice  of  the 
tasks. 

•  Standardized  feedback  to  correct  and  reinforce 
performance  on  the  selected  tasks . 

•  Exercise  support  by  means  of  comprehensive  training 
material . 

In  structured  training,  the  AAR  is  more  than  a  mechanism 
for  delivering  feedback.  It  provides  the  foundation  for  the 
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development  of  structured  training  scenarios.  The  design  of 
structured  training  begins  with  the  specification  of  training 
objectives  (tasks  and  standards)  to  be  covered  in  the  AAR.  The 
simulation  exercises  are  then  carefully  designed  to  ensure  that 
they  provide  the  opportunity  for  units  to  perform  those  tasks 
and  for  the  0/Cs  to  observe  task  performance  (ARI,  1996).  Long 
exercises  are  carefully  divided  into  short  segments  to  allow  for 
frequent  AARs  (Bessemer  &  Myers,  1998) .  The  fact  that 
structured  training  exercises  are  more  controlled  allows  the 
AARs  to  be  more  "focused"  (i.e.,  limited  to  the  specific 
training  objectives  that  define  the  exercise).  With  this 
structure,  the  AAR  leader  can  better  anticipate  whether  and  when 
key  exercise  events  will  occur  and  more  quickly  prepare 
appropriate  AAR  materials  (Meliza,  1996) .  The  standardization 
of  scenarios  and  supporting  materials  also  makes  possible  the 
development  of  AAR  materials  that  can  be  stored  on-line  and 
reused  as  appropriate  (Meliza  &  Paz,  1996) . 

In  1993,  ARI  initiated  the  development  of  materials  and 
methods  for  the  first  such  structured  training  program.  The 
original  training  materials  were  named  Simulation-Based 
Multiechelon  Training  for  Armor  Units  (SIMUTA) .  SIMUTA 
comprised  an  integrated  set  of  platoon  and  company  exercises 
(called  "tables")  for  implementation  on  SIMNET  and  battalion 
staff  exercises  for  implementation  on  Janus,  which  is  a 
constructive  simulation  for  battle  staff  training.  SIMUTA 
materials,  developed  specifically  for  Army  National  Guard  (ARNG) 
battalions,  provided  a  "turn-key"  package  of  items  required  to 
execute  training,  including  orders,  overlays,  and  videotape 
demonstrations  of  successful  performance  (Hoffman,  Graves, 

Roger,  Flynn,  &  Sever,  1995) . 

SIMUTA  methods  and  materials  were  implemented  in  a  program 
managed  by  the  U.S.  Army  Armor  School.  This  program  was  called 
the  Reserve  Component  Virtual  Training  Program  (RCVTP) .  A  key 
component  of  this  program  was  a  dedicated  team  of  0/Cs  whose 
role  was  to  guide  performance  during  the  exercise  and  provide 
feedback  during  the  AAR.  According  to  SIMUTA  guidance,  AARs 
should  occur  frequently  (about  one  every  2  hours  or  at  the  end 
of  every  table)  and  should  be  conducted  about  10-15  minutes 
after  exercise  completion  (ARI,  1996) .  AARs  should  include 
replays  from  the  stealth  (i.e.,  virtual)  display  of  SIMNET  and 
recordings  of  selected  radio  messages.  Shlechter,  Bessemer, 
Nesselroade,  and  Anthony  (1995)  investigated  the  effectiveness 
of  the  RCVTP  and  concluded  that  .it  was  an  effective  program  for 
training  ARNG  units. 
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The  concepts  of  structured  training  and  focused  AARs  have 
been  adopted  in  subsequent  training  programs  developed  by  ARI 
and  implemented  at  the  RCVTP,  which  was  subsequently  renamed  the 
Virtual  Training  Program  (VTP)  to  incorporate  active  and  reserve 
component  units  (Burnside,  Leppert,  &  Myers,  1996) .  In  addition 
to  SIMUTA,  the  additional  major  programs  include  the  following: 

•  The  Simulation-Based  Mounted  Brigade  Training  (SIMBART) 
program  developed  in  1994  for  training  heavy  combat 
brigades  in  the  ARNG  (Roger  et  al .  ,  1996). 

•  The  series  of  efforts  entitled  Combined  Arms  Operations 
at  Brigade  Level  Realistically  Achieved  Through 
Simulation  (COBRAS)  developed  in  1995  for  training 
active  component  brigades  (R.  C.  Campbell  et  al.,  1999) . 

•  The  Staff  Group  Trainer  (SGT)  training  support  packages 
for  battalion  and  brigade  staffs  (Roger  et  al .  ,  1998). 

In  addition,  ARI  has  explored  the  impact  of  the  digitization  of 
C^I  systems  on  structured  training  and  on  the  AAR  process 
(Winsch,  Garth,  Ainslie,  &  Castleberry,  1996) . 

Structured  training  concepts  have  also  been  applied  to  the 
CCTT,  which  is  the  successor  to  SIMNET  (Deatz  et  al . ,  1998).  In 
particular,  ARI  is  developing  the  Commander's  Integrated 
Training  Tool  (CITT)  to  provide  CCTT  users  a  tool  for  selecting, 
modifying,  and  developing  structured  training  exercises  for 
platoons  and  company  teams.  The  CITT  will  provide  the 
capability  to  access,  modify,  and  print  all  required  support 
materials.  Two  versions  of  CITT  are  currently  being  designed: 
a  standalone  version  and  a  distributed  version  that  will  be 
implemented  on  the  Internet  (Gossman,  Dannemiller,  &  Bessemer, 
1998)  . 

Standard  Army  After  Action  Review  System  (STAARS) 

As  discussed  previously,  several  AAR  methods  have  been 
derived  to  fit  particular  training  situations.  Despite  the 
variety  of  authoring  sources  and  training  environments,  Cameron, 
Gentner,  Schopper,  and  Mahaney  (1997)  observed  that  "... 
there  is  a  solid  core  of  agreement  about  what  an  AAR  is,  about 
its  purpose  and  structure,  and  about  how  it  should  be  conducted 
,  .  ."  (p.  5) .  Even  with  this  tacit  agreement,  however,  there 

were  concerns  that  the  format  and  content  of  AAR  data  were  not 
commensurable  across  these  and  other  applications. 
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The  Army  National  Simulation  Center  (NSC)  initiated  STAARS 
in  1995  to  develop  a  structured  and  standard  AAR  process. 

STAARS  was  intended  to  produce  AAR  methods,  data,  and  displays 
that  are  standardized  across  simulation  environments  (live, 
virtual,  and  constructive)  and  across  application  domains 
(training  exercises  and  military  operations  [TEMO] ;  research, 
development,  and  acquisition  [RDA] ;  and  advanced  concepts  and 
requirements  [ACR] ) .  In  1996,  STAARS  came  under  the  aegis  of 
the  Warfighter  XXI  (WF  XXI)  program.  As  such,  it  became  one  of 
the  five  major  components  of  the  WF  XXI  program,  providing  the 
needed  performance  feedback  loop.^ 

Given  ARI's  long  history  of  adapting  the  AAR  process  to 
specific  training  situations,  its  support  of  standardization 
efforts  may  seem  odd,  unless  one  considers  the  benefits  to  be 
gained  by  appropriate  forms  of  AAR  standardization  (Meliza, 

1999) .  When  correctly  applied,  the  concept  of  standardizing  AAR 
aids  can  reduce  the  amount  of  work  trainers  must  perform  in 
planning,  preparing  and  conducting  effective  AARs . 
Standardization  of  AAR  aids  can  also  (a)  reduce  substantially 
the  costs  of  developing  and  maintaining  automated  AAR  systems 
across  training  environments  and  audiences  and  (b)  make  it 
easier  to  link  training  conducted  within  a  wide  variety  of 
settings  with  capstone  training  events,  like  rotations  to  a 
combat  training  center  CTC. 

ARI  has  been  actively  involved  in  the  development  of 
STAARS,  including  providing  input  to  the  STAARS  Operational 
Requirements  Document  (ORD)  and  STAARS  Action  Plan  (Meliza, 

1996).  Many  of  ARI's  recommendations  concern  the  capabilities 
that  an  AAR  system  must  have  to  support  the  timely  preparation 
and  efficient  delivery  of  AAR  aids  in  a  manner  that  keeps 
trainer  workloads  at  a  reasonable  level  and  provides  users  with 


3  The  other  four  components  of  WF  XXI  are  (1)  the  Standard 
Army  Training  System  (SATS) ,  a  computer-based  software  system 
that  integrates  and  automates  the  Army's  training  management 
doctrine;  (2)  Training  Support  Packages  (TSPs) ,  an  integrated 
and  stand-alone  set  of  products,  materials,  and  instructions 
needed  to  plan,  prepare  and,  execute  simulation-based  training; 
(3)  Training  Aids,  Devices,  Simulations,  and  Simulators  (TADSS) , 
technology-based  tools  for  training  soldiers  and  units  to 
prescribed  standards  in  a  resource -effective  manner;  and  (4)  the 
General  Dennis  J.  Reimer  Army  Training  and  Doctrine  Digital 
Library,  a  globally  accessible,  digital  repository  of  training 
knowledge  sets  and  interactive  applications  to  support  training 
of  individuals  and  units. 
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the  flexibility  to  refine  products  in  response  to  lessons 
learned  regarding  their  utility.  These  recommendations  were 
largely  based  on  lessons  learned  from  the  UPAS  and  ATAFS 
projects . 


Current  Status  and  Future  Extensions 

Over  the  last  decade  and  a  half,  the  AAR  process  has  become 
institutionalized  in  Army  training  and  provides  the  focus  for 
virtually  all  collective  training  (Lockheed  Martin  Federal 
Systems,  and  Illusions,  Inc.,  1996) .  The  high  regard  for  the 
AAR  process  is  attested  by  the  following  recent  sample  of 
testimonials : 

After  Action  Reviews  (AARs)  have  proven  to  be  the 
single  most  important  event  in  collective  training. 

This  trend  will  continue  as  we  progress  into  the  21®*^ 

Century  (U.S.  Army  Training  and  Doctrine  Command, 

1997,  p.  1) . 

The  Army's  institutionalization  of  the  AAR  as  an 
essential  part  of  training  is  one  of  the  most 
important  training  interventions  ever.  ...  At  all 
levels,  the  AAR  provides  us  an  honest  appraisal  of  our 
performance  and  directs  our  efforts  to  correct 
shortcomings  (Sullivan,  1995,  pp.  70,  163). 

Excellent  analytical  AARs  during  well  organised  and 
supported  force-on-force  scenarios  against  a  highly 
capable  OPFOR  is  the  secret  of  the  U.S.  Army  recovery 
since  Vietnam  (Hoare,  1996,  p.  17) . 

Another  testimony  to  the  usefulness  and  generality  of  the 
AAR  concept  is  that  institutions  outside  of  the  U.S.  Army. have 
adopted  it.  Experts  from  the  Training  and  Exercise  Division  of  the 
Joint  Warfighting  Center  (JWFC)  conduct  AARs  for  U.S.  Joint  Service 
Elements  ) MacDonald,  1998) .  AAR  capabilities  are  also  incorporated 
into  two  Joint  Service  training  simulations;  the  Joint  Countermine 
Operational  Simulation  (JCOS)  and  the  Joint  Simulation  System 
(JSIMS) .  Foreign  militaries,  including  the  militaries  of  the 
former  Soviet  Union,  have  also  used  the  AAR  (Pitts,  1997) . 

Finally,  the  AAR  has  been  cited  as  a  military  innovation  that 
commercial  organizations  should  adopt  to  provide  feedback  and 
effect  positive  change  (Davenport  &  Prusak,  1997;  Sullivan  & 
Harper,  1996. 
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Conceptual  Foundations 

One  of  the  defining  characteristics  of  the  AAR  process  is 
that  it  is  expressly  based  on  behavioral  science  principles. 
Because  the  AAR  is  a  multi-faceted  process,  these  principles 
have  been  drawn  from  a  variety  of  behavioral  science 
disciplines.  The  purpose  of  this  section  is  to  identify  and 
document  the  behavioral  science  theories  and  principles  that 
provide  the  foundations  of  the  AAR  process.  We  also  review 
recent  AAR  research  that  is  relevant  to  those  principles. 

Identifying  behavioral  science  principles  was  not  always  a 
straightforward  process.  In  some  cases,  these  principles  were 
explicitly  stated  in  the  AAR  research  literature.  In  other 
cases,  these  principles  were  inferred  from  the  guidance 
provided.  Also,  certain  prescriptions  were  not  unique  to 
particular  theories.  For  instance,  the  AARs  emphasis  on  active 
learning  can  be  derived  from  various  points  of  view.  The  reader 
should  keep  in  mind  that  the  purpose  of  the  analysis  was  not  to 
provide  a  definitive  and  exhaustive  listing  of  all  behavioral 
science  principles  that  have  guided  the  development  of  the  AAR 
process.  Rather,  the  analysis  was  intended  to  provide  an 
appreciation  for  the  range  and  variety  of  the  theoretical 
concepts  that  have  been  incorporated  into  the  AAR. 

Table  3  presents  the  results  of  this  analysis.  This  table 
lists  the  theories  and  techniques  that  have  led  to  methods  or 
products  that  have  been  incorporated  into  the  AAR  process.  For 
expository  purposes,  these  specific  theories  and  techniques  are 
grouped  into  six  general  areas  of  theory  and  research: 

1.  Information  feedback. 

2 .  Performance  measurement . 

3.  Memory  and  cognition. 

4.  Group  processes  and  dynamics. 

5.  Communication  theory  and  techniques. 

6.  Instructional  science. 

Each  area  is  divided  into  several  specific  theories  or  methods 
that  have  given  rise  to  AAR  methods,  practices,  or  products. 
These  areas  were  then  used  as  topics  for  the  more  detailed 
analysis, of  current  research  issues. 
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Table  3 

AAR  Methods,  Practices,  and  Products  Developed  from  Behavioral 
Science  Principles 


Theories  and 

Research  Areas _ Techniques 


AAR  Methods,  Practices,  or  Products 


Information  Intrinsic  Development  of  live  simulation  systems (SCOPES, 

Feedback  feedback  REALTRAIN,  MILES)  to  provide  realistic  EDA 

Use  of  AAR  to  determine  fidelity  requirements 

Live  feedback  requirements  for  emerging  weapon 
systems 

Extrinsic  Design  that  minimizes  delay  of  feedback 

feedback 

Improvements  to  take-home  package  (THP) 

Intervention  guidelines  for  providing  coaching  and 
mentoring 

Performance  Process  vs.  Incorporation  of  process  with  product  measures  to 

Measurement  product  aid  diagnosis  of  performance  problems 

measurement 

Automated  Unit  Performance  Assessment  System  (UPAS) 

performance 

measurement  Automated  Training  Analysis  and  Feedback  System 

technology  (ATAFS) 

Standardization  of  AAR  through  input  to  STAARS 

Self-assessment  Experimental  methods  for  providing  feedback  for 

techniques  multiforce  exercises 

Memory  and  Transfer  of  Focus  on  tactics,  not  gaming 

Cognition  training 

Memory  aiding  Initial  review  of  "what  happened"  to  refresh 

participants'  memories 

Use  of  summaries  to  reinforce  AAR  points 
THPs  to  refresh  memory  after  AAR 

Problem-solving/  Determination  of  causes  of  performance  problems 

decision-making  ■  (why) 

Determination  of  training  solutions  (how  to 
improve) 

Mental  models  Benefits  of  multiple  points  of  view 

Participation  of  OPFOR 

(table  continues) 
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Research  Areas 


Theories  and 
Techniques 


AAR  Methods,  Practices,  or  Products 


Social  Exploiting  the  positive  effects  of  group  context 

facilitation  and 

social  loafing  Minimizing  irrelevant  distractions  during  AAR 

Monitoring  participation  to  prevent  "social 
loafing" 

Group  identity  Establishing  positive,  non- threatening  atmosphere 
and  cohesiveness 

Minimizing  "finger-pointing" 

AAR  leader  acting  as  moderator,  not  discussant 

Having  unit  identify  problems  and  provide  solutions 

Communication  Descriptive  Use  of  specific  statements  instead  of  abstractions 

Theory  and  communication 

Techniques  References  to  task  goals  and  objectives 

Questioning  Emphasis  on  open-ended  questioning 

techniques 

Form  and  content  Prescriptive  model  of  feedback  that  stresses 
of  feedback  •  Performance,  not  personal  characteristics 

•  Importance  of  rationale 

•  References  to  goals  and  objectives 

•  Strategies  for  improving  performance 

Instructional  Guided  discovery  Active  identification  and  solution  of  problems  by 

Science  learning  unit  members 

Unit  control  of  AAR  content,  with  AAR  leader 
guiding  the  process 

Active  learning  in  a  realistic  group  context 

Importance  of  iterative  cycles  of  exercises  and 
AARs 

Cooperative  Encouragement  of  group  participation  in  discussions 

learning 

Systems  concepts  Analysis  of  AAR  processes  and  subprocesses 


Experiential 

learning 


Group 

Processes  and 
Dynamics 
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Information  Feedback 


Perhaps  the  most  basic  principle  on  which  the  AAR  is  based 
is  that  learning  and  performance  are  enhanced  when  appropriate 
feedback  is  provided.  Feedback  refers  to  information  that 
people  receive  during  or  after  performance  of  an  action  to 
control  and  learn  the  action.  As  discussed  below,  the  feedback 
can  come  from  different  sources.  A  fundamental  distinction  in 
types  of  sources  for  information  feedback  is  that  between 
intrinsic  and  extrinsic  feedback.  Intrinsic  feedback  refers  to 
information  that  is  inherent  to  task  performance,  whereas 
extrinsic  feedback  denotes  information  that  augments  or 
supplements  inherent  feedback  (Winstein  &  Schmidt,  1989) . 

Intrinsic  Feedback 


In  the  context  of  tactical  training.  Brown,  Nordyke,  et  al . 
(1998)  defined  intrinsic  feedback  as  the  "downrange"  information 
provided  to  collective  training  participants  from  both  real  and 
simulated  sources.  The  simulated  outcomes  from  tactical 
engagements  are  important  sources  of  intrinsic  feedback, 
providing  information  necessary  for  learning  during  the  battle. 
This  source  of  feedback  is  necessary  to  cue  and  guide 
performance  during  exercises,  providing  an  opportunity  for 
meaningful  practice  of  task  performance.  For  example,  if  a  unit 
is  receiving  no  feedback  during  exercises  about  the  location  and 
effects  of  its  artillery  missions,  the  unit  cannot  meaningfully 
perform  tasks  associated  with  adjusting  and  terminating  the  use 
of  artillery.  After  the  battle,  exercise  participants  use  their 
memory  of  this  feedback  to  answer  the  first  and  fundamental 
issue  of  the  AAR:  to  establish  "what  happened"  during  the 
battle. 

One  of  the  most  frequently  asked  questions  in  an  AAR  is  why 
a  unit  chose  a  particular  course  of  action  (e.g..  Why  did  you 
decide  to  open  fire  as  soon  as  you  saw  the  enemy?")  The 
question  sets  the  stage  for  ensuring  that  the  unit  does  what  it 
should  do  to  collect  and  interpret  information  about  the 
tactical  situation  and  then  select  an  appropriate  course  of 
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action.  If  there  are  problems  with  the  quality  of  intrinsic 
feedback  provided  to  units,  artificial  aspects  of  the  training 
situation  may  guide  unit  decisions  or  the  training  environment 
may  not  allow  the  unit  to  employ  an  appropriate  course  of 
action.  The  following  sections  provide  a  discussion  of  ARI's 
involvement  in  issues  related  to  intrinsic  feedback,  first  with 
regard  to  the  fidelity  of  virtual  simulations  and  then  with 
regard  to  requirements  of  live  simulations. 

Simulator  Fidelity.  With  regard  to  intrinsic  feedback,  one 
issue  that  continues  to  interest  researchers  concerns  simulator 
fidelity  (i.e.,  the  correspondence  of  stimuli  and  responses 
between  simulators  and  the  actual  equipment  that  they  emulate) . 
Many  simulator  designers  have  implicitly  endorsed  what  may  be 
termed  a  "full  fidelity"  model,  wherein  the  simulator  is 
designed  to  emulate  as  much  of  the  physical  system  as  possible. 
In  contrast,  SIMNET  developers  introduced  the  concept  of 
"selective  fidelity,"  which  stipulates  that  the  simulation 
should  replicate  only  those  aspects  of  the  physical  system  that 
contribute  directly  to  training.  Those  aspects  of  the  physical 
system  that  do  not  contribute  directly  are  represented  in  low 
fidelity  or  are  not  represented  at  all  (Thorpe,  1987) . 

ARI  developed  a  method  for  rationally  determining  the 
fidelity  of  SIMNET  after  it  had  been  fielded  (Burnside,  1990) . 
Burnside  had  subject  matter  experts  rate — according  to  explicit 
criteria — the  degree  to  which  tasks  could  be  performed  within 
the  SIMNET  environment,  using  a  3 -point  scale:  fully  trained, 
partially  trained,  and  not  trained.  Assuming  that  tasks  must  be 
at  least  partially  performable  to  be  trainable  in  SIMNET,  his 
results  indicated  that  35  percent  of  tasks  from  the  Army 
Training  and  Evaluation  Program  (ARTEP)  Mission  Training  Plan 
(MTP)  could  be  trained  with  SIMNET.  The  platoon  echelon  had  the 
highest  percentage  of  trainable  tasks  (41  percent)  and  of  tasks 
not  supported  by  the  simulation  (46  percent) .  On  the  basis  of 
these  findings,  Burnside  (1990)  concluded  that  this  method  was 
an  efficient  analytic  approach  to  determining  the  fidelity  of 
any  training  aid,  device,  simulation,  or  simulator. 

Subsequently,  these  methods  were  used  to  identify  appropriate 
training  objectives  for,  and  feedback  from,  the  structured 
training  exercises  in  SIMUTA  (Hoffman  et  al.,  1995). 

Behavior  of  Computer-Generated  Forces  (CGF) .  Another  major 
fidelity  issue  with  which  ARI  has  been  involved  is  the  realism 
of  the  behavior  of  friendly  and  enemy  computer  generated  forces 
(CGF) .  For  example,  Meliza  and  Vaden  (1995)  compared  two 
versions  of  software  that  controlled  the  behavior  of  the  CGF  in 
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SIMNET:  Modular  Semi -automated  Force  (ModSAF)  Version  1.2  and 

Semi -automated  Force  (SAFOR)  Version  4.1.3.  One  of  the 
comparisons  involved  using  the  UPAS  to  examine  movement  speeds 
of  tanks  generated  by  each  version  of  CGF  as  the  tanks  moved 
along  the  same  route  to  decide  if  one  was  more  sensitive  to 
slope  and  other  terrain  variables  that  should  influence  speed. 
The  SAFOR-controlled  tank  drove  at  the  same  speed  for  98  percent 
of  the  test  period,  while  the  ModSAF- controlled  tank  drove  at  a 
wide  variety  of  speeds . 

Exercise  Control  for  Live  Simulations.  Trainers  for  live 
force-on- force  face  a  heavy  workload.  In  addition  to  coaching 
and  mentoring  units  and  conducting  AARs,  they  must  often  help 
simulate  the  effects  of  weapons  and  play  the  role  of  higher, 
adjacent,  and  supporting  units  so  that  the  units  they  are 
training  will  receive  appropriate  intrinsic  feedback.  The 
duties  associated  with  providing  this  feedback  can  detract  from 
the  time  available  to  prepare  for  AARs. 

The  Army  is  fielding  a  large  variety  of  new  and  innovative 
operational  systems  over  the  next  10  years  under  the  "force 
modernization"  and  "Force  XXI"  rubrics,  including  new  weapon 
systems  (smart  and  non- lethal  varieties) ;  reconnaissance, 
surveillance,  and  target  acquisition  (RSTA)  systems;  and  digital 
communication  and  decision  aid  systems.  Training  developers 
have  suspected  that  the  feedback  requirements  for  these  new 
systems  will  be  significantly  greater  than  those  for  extant 
systems  and  could  overwhelm  the  trainers . 

In  response  to  a  studies  request  from  the  TRADOC  Army 
Training  Modernization  Directorate,  ARI  initiated  a  program  in 
February  1997  to  study  instrumentation  issues  relevant  to  the 
support  of  AARs  in  the  live  simulation  training  environment 
(i.e.,  at  the  CTCs  and  at  home  station  field  exercises) .  The 
result  was  a  document  entitled  "The  Training  Analysis  and 
Feedback  Aids  (TAAF-Aids)  Study"  (Brown,  Nordyke,  et  al . ,  1998). 
The  primary  goal  of  this  effort  was  to  describe  the  impacts  of 
new  weapons,  RSTA,  and  digital  C^I  systems  on  exercise  and 
feedback  functions  performed  by  0/Cs  and  analysts  who  support 
training.  This  work  also  set  the  stage  for  the  longer  term 
objective  of  applying  automation  to  support  the  training 
feedback  process.  In  that  regard,  the  secondary  objective  was 
to  present  high-level  strategies  for  helping  trainers  address 
the  increasing  workload  expected  to  be  required  by  the  new 
systems . 
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The  TAAF-Aids  Study  examined  the  feedback  requirements  of 
142  new  systems  in  live,  force-on-force  training.  Based  on  the 
capabilities  of  current  TES  and  instrumentation  systems,  this 
study  estimated  whether  specific  elements  of  feedback  would  be 
provided  by  either  the  TES- instrumentation  system,  by  0/Cs  and 
the  analysts  who  support  them,  or  through  soldier  interaction 
with  the  operational  system.  Of  particular  interest  were  those 
cases  where  current  TES-instrumentation  systems  fail  to  provide 
appropriate  intrinsic  feedback,  requiring  the  0/C-analyst  team 
to  affect  or  control  the  exercise  to  provide  the  needed 
feedback.'^  On  the  basis  of  their  findings.  Brown,  Nordyke,  et 
al.  (1998)  concluded  that,  in  the  absence  of  effective 
interventions,  the  0/Cs,  and  analysts  will  be  overwhelmed  by  the 
need  to  monitor  and  control  the  simulation  in  order  to  provide 
feedback. 

Figure  3  shows  how  the  intrinsic  feedback  needed  to  cue  and 
guide  the  employment  of  one  of  the  new  systems  during  unit 
exercises  can  come  from  interaction  with  equipment,  TES  or 
instrumentation  systems,  and  trainers.  In  certain  cases,  there 
may  even  be  gaps  in  feedback.  For  cases  where  it  appears  that 
trainers  will  be  responsible  for  providing  feedback,  the 
exercise  control  and  functions  required  to  provide  feedback  were 
identified  in  the  TAAF  Aids  Study,  as  shown  in  Figure  4.  An 
important  Army  goal  is  to  apply  automation  (TES  systems  and 
instrumentation)  to  free  trainers  from  supporting  simulations  so 
that  they  can  spend  more  time  coaching,  mentoring,  and  preparing 
for  AARs.  In  deciding  where  to  apply  automation,  the  Army  must 
consider  that  some  of  the  exercise  control  functions  directly 
support  coaching,  mentoring,  and  AARs,  while  others  distract 
trainers  from  these  activities. 


^  Brown  et  al .  (1998)  studied  the  effects  of  modernization  on 

extrinsic  as  well  as  intrinsic  feedback.  They  determined  that 
the  control  tasks  they  identified  as  required  for  intrinsic 
feedback  also  supported  extrinsic  feedback. 
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Figure  3.  Sources  of  intrinsic  feedback  needed  by  the 
gun  crew,  fire  direction  center,  laser  designator 
operator,  and  target  vehicle  crew  relative  to  employment 
of  a  120-MM  precision-guided  mortar  munition. 
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1.  Monitor  Observer  Call  for  Fire 
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2.  Assess  Call  for  Fire  and  Laser 
designation  procedures 

3.  If  procedures  valid  call  TAF  to 
notify  OC  closest  to  victim(s),  and 
tell  them  to  administratively  kill 
vehicle(s)  depending  on  number 
of  rounds  fired 


FDC  OC 


1.  Receive  Fire  Mission  Data,  from  FDC  personnel, 
forward  target  location  for  TAF  analyst  to  enter 
Into  TAF  workstation 

2.  Receive  PLT  OC  report  on  validity  of  Gun 
Section  procedures 

3.  Coordinate  with  FO/COLT  OC  on  validity  of  FO 
procedures 

4.  If  FO/COLT,  FDC,  and  Gun  section  procedures 
valid  tell  TAF  analyst  to  fire  mission 


Figure  4.  Examples  of  trainer  exercise  control  and 
feedback  functions  involved  in  providing  intrinsic 
feedback. 
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Extrinsic  Feedback 


In  collective  training,  extrinsic  feedback  refers  to 
information  provided  in  the  AAR,  coaching  and  mentoring  during 
the  exercise,  and  information  in  the  take-home  package  (THP) 
given  to  units  after  a  series  of  exercises.  Each  of  these 
sources  of  extrinsic  feedback  is  discussed  below. 

Information  Provided  in  the  After  Action  Review  (AAR) .  The 
information  provided  in  the  AAR  is  a  form  of  extrinsic  feedback  ' 
that  is  sometimes  referred  to  as  "knowledge  of  results"  or  KR. 

KR  has  been  repeatedly  demonstrated  to  be  one  of  the  most 
powerful  variables  in  the  acquisition  and  retention  of  simple 
and  complex  tasks.  In  general,  any  manipulation  that  increases 
the  quantity  or  quality  of  KR  has  a  positive  effect  on  learning 
and  performance.  According  to  this  principle,  then,  AARs  should 
be  scheduled  as  often  as  possible,  usually  immediately  after 
each  exercise  or  simulated  mission.  Because  much  of  the 
discussion  during  the  AAR  depends  on  the  participants'  memories 
for  events  and  outcomes  that  occurred  during  the  exercise,  the 
AAR  is  usually  scheduled  as  soon  as  possible  after  ENDEX  to 
mitigate  the  effects  of  forgetting.  To  derive  maximum  benefit 
from  the  AAR,  subsequent  exercises  must  be  scheduled  soon  after 
the  end  of  the  AAR  so  participants  can  implement  lessons  learned 
from  their  feedback.  Accordingly,  the  tactical  training  tables 
and  exercises  developed  for  the  VTP  are  designed  to  minimize  two 
intervals:  the  interval  between  the  original  exercise  and  the 
AAR  session  and  the  one  between  the  AAR  session  and  the  next 
exercise  (Hoffman  et  al . ,  1995). 

Coaching  and  Mentoring.  For  the  most  part,  0/Cs  interfere 
little  in  simulation  exercises  and  allow  participants  to 
experience  the  consequences  of  their  own  decisions  and 
responses.  On  occasion,  however,  0/Cs  may  need  to  intervene  for 
reasons  such  as  training  an  inexperienced  crew,  advancing  the 
action,  or  preventing  counterproductive  actions.  Research 
suggests  that  providing  feedback  during  these  interventions  can 
be  effective  under  certain  circumstances;  however,  it  must  be 
used  judiciously  to  prevent  performers  from  developing  a 
dependence  on  extrinsic  feedback  and  to  encourage  the  processing 
of  more  appropriate  intrinsic  feedback  sources.  The  VTP, 
recognizing  the  need  for  occasional  coaching  while  also  knowing 
its  potential  problems,  drafted  and  tested  "intervention 
guidelines"  for  instructing  0/Cs  on  when  and  how  to  intervene 
during  a  SIMUTA  exercise  (Hoffman  et  al . ,  1995). 
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Take-Home  Packages  (THPs) .  Important  sources  of  feedback 
during  the  AAR  are  the  discussions  on  how  to  sustain  performance 
strengths  and  improve  on  weaknesses  observed  during  the 
simulated  exercise.  The  purpose  of  the  THP  is  to  document  these 
lessons  learned  so  that  they  can  be  implemented  at  home  station. 

ARI  investigated  the  extent  to  which  THP  feedback  is 
actually  used  by  interviewing  the  leaders  of  units  that  had 
recently  trained  at  NTC  (Fobes  &  Meliza,  1989) .  Fobes  and 
Meliza  found  leaders  to  be  generally  receptive  to  NTC  feedback. 
In  particular,  unit  leaders  felt  that  information  from  the  NTC 
should  be  used  "...  to  improve  unit  SOPs,  refine  their  Tables 
of  Organization  and  Equipment,  and  address  gaps  in  unit  training 
plans"  (p .  13).  However,  these  same  leaders  criticized  the 
print -based  parts  of  the  THP  for  being  too  lengthy  and  for  being 
filled  with  unnecessary,  irrelevant,  and  sometimes  inconsistent 
details.  Most  important,  they  thought  that  the  paper-based 
parts  of  the  THP  failed  to  provide  concrete  solutions  for  their 
training  problems.  On  the  other  hand,  the  interviewees  found 
that  the  videotaped  AARs,  which  were  part  of  the  THP,  provided 
focused  and  specific  information  that  could  be  used  to  implement 
positive  innovations.  Fobes  and  Meliza  used  information  from 
these  interviews  to  provide  specific  suggestions  on  the 
structure  and  content  of  THPs. 


Fober,  Dyer,  and  Salter  (1994)  found  similar  results  in 
their  analysis  of  THPs  from  units  that  rotated  through  the  Joint 
Readiness  Training  Center  (JRTC) .  The  JRTC  is  similar  in 
concept  and  purpose  to  the  NTC  but  focuses  on  light  (as  opposed 
to  heavy)  combat  units.  They  performed  a  content  analysis  of 
THPs  from  45  companies  and  15  infantry  battalions  that  rotated 
through  the  JRTC  in  Fiscal  Year  (FY)  1990.  The  purpose  of  their 
analysis  was  to  determine  the  value  of  THPs  as  a  source  of 

measures.  Overall,  they  found  that  THPs  provided 
detailed  information  on  unit  strengths  and  weaknesses  that  is 
valuable  both  as  feedback  to  the  unit  and  as  performance  data 
for  the  researcher.  However,  they  also  noted  some  of  the 
weaknesses  of  THPs  as  performance  measures.  In  particular,  they 
examined  the  training  recommendations  in  THPs  as  a  method  to 
compare  unit  performance.  Their  analysis  indicated  that  the 
training  recommendations  in  the  THPs  tended  be  boiler  plated 
i^fo^^stion  (i.e.,  that  similar  training  advice  is  provided  for 
all  THPs) .  For  example,  many  THPs  included  the  recommendation 
conduct  more  "force-on-force,  multi-echelon  training."  Although 
such  information  is  based  on  actual  perceived  weaknesses,  the 
generic  nature  of  the  feedback  makes  it  difficult  to  implement 
in  an  actual  training  plan. 
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Performance  Measurement 


The  AAR  process  is  based  on  the  objective  measurement  of 
performance,  by  either  a  trained  observer  or  an  automated 
system.  In  this  section,  we  discuss  four  performance 
measurement  issues  that  have  a  direct  impact  on  the  AAR  process: 

1.  Product  and  process  performance  measurement. 

2.  Data  stream  issues. 

3 .  Automated  performance  measurement  systems . 

4.  Self-assessment  techniques. 

The  first  issue  is  primarily  theoretical,  concerning  the  types 
of  performance  measures  that  can  and  should  be  obtained  for 
feedback  purposes.  The  next  two  issues  are  essentially 
technological,  relating  to  the  nature  of  the  data  stream  in 
Distributed  Interactive  Simulation  (DIS) -based  simulation 
systems  and  to  the  automation  of  performance  measurement 
functions.  Finally,  the  last  issue  addresses  a  different 
approach  to  performance  measurement — an  approach  that  is  based 
on  self-assessment. 

Product  and  Process  Performance  Measures 

A  fundamental  distinction  in  performance  measurement  is 
that  between  product  and  process  measures.  Product  measures 
refer  to  the  outcomes  or  end  results  of  performance,  such  as 
number  of  friendly  and  threat  vehicles  lost  during  an 
engagement .  Process  measures  refer  to  the  procedure  or 
technique  used  to  achieve  (or  attempt  to  achieve)  an  outcome, 
such  as  percent  of  tasks  correctly  performed  to  standard.  A 
similar  distinction  is  drawn  in  the  test  and  evaluation  (T&E) 
community  between  measures  of  effectiveness  (battle  outcomes) 
and  measures  of  performance  (process  for  achieving  those 
outcomes) . 

A  primary  rationale  for  developing  TES  methods  in  1970  was 
to  provide  objective  indicators  and  measures  of  battle  outcomes. 
Some  felt  that  these  types  of  product  measures  were  more 
relevant  than  process  measures  and  provided  all  the  information 
that  soldiers  needed.  For  example,  Shriver  et  al.  (1975) 
maintained  that  "...  the  measurement  objective  was  to  record 
data  that  indicated  whether  the  defined  job  had  been 
successfully  performed  .  .  .  rather  than  whether  the  correct 
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processes  or  techniques  had  been  employed  ..."  (p.  5).  This 
same  emphasis  on  product  measurement  was  also  notable  in  the 
early  days  of  SIMNET  development  (e.g.,  Thorpe,  1987). 

Early  on,  however,  ARI  researchers  realized  that  process 
measures  were  necessary  in  AARs  to  identify  causes  of  outcomes 
(i.e.,  to  answer  the  question  "Why  did  it  happen?") .  For 
instance,  significant  differences  between  REALTRAIN  and 
conventionally  trained  units  were  observed  for  several  process- 
oriented  performance  measures  (Meliza,  Scott,  &  Epstein,  1979; 
Scott,  Meliza,  Hardy,  &  Banks,  1979) .  To  explain  how  light 
infantry  squads  approached  and  crossed  danger  areas  during 
movement  to  contact  missions,  ARI  placed  a  data  collector  on  the 
far  side  of  a  danger  area  to  collect  this  information.  In  the 
pre-training  tests  for  both  training  conditions  and  in  the  post¬ 
training  test  for  conventionally  trained  units,  the  data 
collectors  observed  most  members  of  the  squad  before  the  first 
soldier  crossed  the  danger  area  and  usually  heard  the  unit 
approaching  the  danger  area  before  observing  any  soldiers.  For 
the  squads  trained  using  TES  and  AARs,  very  few  soldiers  were 
observed  before  the  first  soldier  crossed  the  danger  area,  and 
the  data  collector  usually  heard  nothing  before  seeing  the  first 
soldier.  The  well-trained  squads  made  contact  with  the  enemy  in 
a  manner  that  caused  minimum  risk  to  the  squad  as  a  whole, 
helping  to  explain  why  these  units  demonstrated  a  higher  rate  of 
mission  success,  inflicted  more  casualties  on  the  enemy,  and 
sustained  fewer  casualties. 

Training  programs  that  were  developed  later  have  provided 
for  the  collection  of  product  and  process  performance  measures. 
For  instance,  the  AARs  in  the  VTP  used  outcome  data  from  UPAS 
and  process  measures  of  unit  performance  collected  by  0/Cs 
(Hoffman  et  al . ,  1995).  The  present  consensus  is  that  both 
types  of  information  are  useful  in  AARs;  product  measures  appear 
most  helpful  in  answering  the  first  AAR  question  (What 
happened?) ,  whereas  process  measures  aid  participants  in 
answering  the  second  AAR  question  (Why  did  it  happen?) . 

Feedback  can  be  particularly  effective  when  those  two  types  of 
data  are  integrated  to  provide  a  more  complete  understanding  of 
battle  events.  An  example  of  integrated  AAR  display  is  the 
Exercise  Timeline  incorporated  in  the  UPAS  (Meliza,  Bessemer,  & 
Tan,  1994) .  This  display  provides  information  on  product 
measures  (e.g.,  enemy  vehicles  destroyed)  along  with  crew 
processes  (e.g.,  radio  reports)  depicted  on  a  common  timeline. 
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Data  Stream  Issues 


The  potential  for  automating  performance  measurement 
functions  was  greatly  enhanced  with  the  introduction  of 
computer-based  simulations,  which  provide  rich  data  streams  for 
analysis  and  feedback.  However,  the  technology  for  measuring 
performance  on  such  systems  is  complex  and  is  constantly 
evolving.  ARI  has  continued  to  monitor  this  technological  issue 
through  the  Workshops  on  Standards  for  the  Interoperability  of 
Defense  Simulations,  where  standards  are  written  and  monitored 
for  the  DIS  protocols  that  replaced  SIMNET  protocols.  These 
standards  are  being  developed  for  all  U.S.  Services,  allied 
military  units,  and  industry  with  the  intent  of  supporting 
exercises  in  the  live,  virtual,  constructive,  and  mixed 
environments . 

An  example  data  stream  issue  that  impacts  AAR  is  the  Change 
of  Status  (COS)  data  packet.  ARI  requested  a  DIS  data  packet 
that  would  serve  the  same  function  that  the  COS  data  packet  had 
served  in  SIMNET.  A  simulated  entity  sends  this  packet  over  the 
network  when  the  entity  is  tactically  damaged  or  destroyed, 
administratively  destroyed,  or  administratively  reincarnated. 
This  packet  identifies  the  nature  of  the  status  change  and  the 
cause  of  the  status  change.  Under  the  original  DIS  protocols, 
there  was  no  direct  replacement  for  the  COS  packet.  As  a 
result,  the  AAR  system  (or  the  human  operator)  was  tasked  with 
deciding  the  cause  of  status  changes  when  preparing  many  data 
summary  graphs  and  tables,  thus  slowing  down  the  preparation  of 
AAR  aids  (Meliza,  1995) . 

Automated  Performance  Measurement  Systems 


Automating  performance  measurement  systems  have  had  a 
positive  effect  on  the  AAR  process  in  at  least  two  ways.  First, 
they  potentially  have  increased  the  amount  and  accuracy  of 
information  available  for  the  AAR.  Second,  they  have  relieved 
0/Cs  from  many  manual  performance  measurement  functions  and  have 
allowed  them  to  focus  on  other  important  functions,  such  as 
observing  global  aspects  of  performance  and  providing  coaching 
and  mentoring  as  appropriate.  As  discussed  in  the  Historical 
Foundations  section,  ARI  directed  the  development  of  two  systems 
for  automating  performance  measurement  functions  in  SIMNET  and 
other  DIS  systems:  UPAS  and  ATAFS .  The  following  sections 
summarize  the  R&D  efforts  that  supported  the  production  of  UPAS 
and  ATAFS.  They  also  include  research  on  STAARS,  an  evolving 
system  that  is  based  on  the  lessons  learned  from  these  two 
systems . 
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Unit  Performance  Assessment  System  (UPAS) .  Meliza, 

Bessemer,  Burnside,  and  Shlechter  (1992)  examined  the  ARTEP  MTP, 
which  describes  the  requirements  for  tactical  training.  From 
these  requirements,  these  researchers  recommended  that  the  UPAS 
Plan  View  Display  be  improved  by  including  the  major  terrain 
features  and  tactical  control  measures.  Some  additional 
recommendations  included  the  ability  to  search  rapidly  through 
the  exercise  file  (as  opposed  to  a  slower  visual  search)  and  the 
ability  to  magnify  the  battlefield  depictions  to  sections  as 
small  as  200  x  200  m.  In  addition,  they  suggested  that  UPAS 
include  the  following  display  formats: 

•  Battle  Flow.  This  is  an  animated  figure  that  traces  the 
movements  of  vehicles  and  units  throughout  the  mission 
or  mission  segment.  Symbols  are  used  to  mark  positions 
at  specified  time  periods  (e.g.,  every  minute).  Unlike 
the  Plan  View  Display,  the  symbols  do  not  indicate 
vehicle  and  weapons  system  orientation.  This  display 
was  primarily  designed  to  provide  feedback  on  movement 
formations  and  techniques. 

•  Battle  Snap  Shot.  This  provides  still  shots  of  vehicle 
positions  in  relation  to  terrain  and  control  measures. 
This  display  includes  icons  that  indicate  vehicle  and 
weapon  system  orientation. 

•  Exercise  Timeline.  This  displays  events  on  three 
timelines  corresponding  to  movement,  direct  and  indirect 
firing,  and  communication.  The  purpose  is  to  provide 
feedback  on  the  coordination  of  these  three  functions. 
Figure  5  presents  a  typical  Exercise  Timeline  Display 
that  can  be  used  at  an  AAR. 
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SIMNET  EXERCISE  TIMELINE 

EXERCISE  ID:  005  DRTE:  S2~8-3  Conpany  C  LEUEL:  PLT  FORCE:  RED 


Shlechter,  Bessemer,  Wade,  and  Nesselroade  (1994)  had 
potential  users  of  UPAS  rate  how  likely  they  were  to  use  each  of 
the  four  display  formats  described  by  Meliza,  Bessemer, 

Burnside,  and  Shlechter  (1992) :  the  Plan  View,  Battle  Flow, 
Battle  Snap  Shot,  and  Exercise  Timeline.  They  also  had  users 
rate  two  additional  display  formats.  Battle  Scorecard  and 
Graphs ,  which  provide  summary  data  in  tabular  and  graphical 
forms,  respectively.  (Figure  6  presents  an  example  of  a  UPAS 
Graph.)  Users  indicated  a  preference  for  the  Battle  Snap  Shot 
Display  and  suggested  that  line-of -sight  vectors  be  added  to 
indicate  clear  visual  paths  and  blind  spots  in  crewmembers' 
fields  of  vision.  Users  also  suggested  the  development  of  a 
Fire  Fight  Display,  which  would  show  the  results  of  selected 
segments  of  the  battle  by  indicating  the  hits,  kills,  and  misses 
for  each  tank  by  displaying  color-coded  vectors  among  all 
vehicles  (live  or  dead)  at  the  end  of  a  designated  time  segment. 
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Figure  6.  Example  of  a  battle  graphics  display  from  UPAS. 

Meliza,  Bessemer,  and  Tan  (1994)  also  documented  a  variety 
of  improvements  to  the  UPAS.  In  addition  to  display 
improvements  described  in  an  earlier  report  (Meliza,  Bessemer, 
Burnside,  &  Shlechter,  1992),  Meliza,  Bessemer,  and  Tan 
discussed  the  following  enhancements  to  UPAS: 

•  Improving  methods  to  select,  filter,  and  process  PDUs 
from  the  network.  The  purpose  was  to  prevent  UPAS  from 
becoming  overloaded  with  data. 

•  Increasing  the  rate  at  which  network  data  are  loaded 
into  a  relational  database.  The  goal  was  to  have  data 
analyzed  10  minutes  after  ENDEX. 

•  Adding  a  time  indexing  file  for  PDUs  for  use  with  all 
map  displays.  This  facility  permits  users  to  move 
guickly  from  one  point  in  the  exercise  to  another. 
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•  Introducing  an  AAR  Presentation  Manager.  This  tool  is 
intended  to  integrate  the  various  UPAS  displays  into  a 
smooth  AAR  presentation. 

Automated  Training  Analysis  and  Feedback  System  (ATAFS) . 
incorporated  four  display  formats  that  were  shown  to  be  useful 
in  UPAS:  the  Plan  View,  Battle  Snap  Shot,  Fire  Fight,  and  Battle 
Flow.  (Figure  7  shows  an  example  of  the  Fire  Fight  Display.) 
ATAFS  also  provided  graphs  and  tables  similar  to  those  used  in 
UPAS.  However,  ATAFS  differed  from  UPAS  in  some  critical  ways. 
First,  ATAFS  was  implemented  on  a  faster  hardware  platform  and 
was  better  able  to  meet  the  minimum  acceptable  delay  in 
preparing  feedback  for  a  SIMNET  AAR.  Second,  the  multi-tasking 
system  and  audio  capabilities  of  ATAFS  allowed  it  to  record  and 
play  back  radio  communications  synchronized  with  battle  events. 
Third,  and  most  important,  ATAFS  incorporated  a  limited  internal 
knowledge  base,  which  was  designed  to  help  the  0/C  choose  and 
create  AAR  displays. 


Figure  7.  Example  of  a  fire  fight  display  from  ATAFS. 
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The  ATAFS  knowledge  base  could  be  used  to  generate  a  sample 
set  of  prototype  displays  that  were  automatically  created  based 
on  performance  elements  of  the  platoon  task  entitled  "Execute 
Actions  on  Contact."  In  addition,  0/Cs  could  manually  generate 
other  aids  to  help  illustrate  points  in  the  AAR.  All  aids  were 
stored  in  a  "bin"  from  which  the  user  selected  and  sequenced  the 
aids  for  presentation.  Figure  8  provides  a  display  of  items  in 
a  typical  ATAFS  bin.  As  indicated  by  the  VCR  button  in  the 
bottom  right  of  this  figure,  ATAFS  also  has  the  capability  to 
load  display  images  onto  videotape.  This  feature  frees  up  the 
ATAFS  workstation  for  subsequent  data  collection  and  creates  a 
tangible  product  that  AAR  participants  can  replay  at  home 
station. 
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Figure  8 .  Example  of  display  showing  items  in  the  ATAFS 
AAR  aid  bin. 

Meliza  and  Brown  (1996)  discussed  improvements  to  the  ATAFS 
system  that  potentially  increase  its  effectiveness  in  providing 
relevant  feedback  to  units.  The  current  ATAFS  is  capable  of 
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generating  a  virtually  unlimited  number  of  "ground  truth"  aids, 
which  are  best  suited  to  answering  the  "what  happened"  and 
perhaps  the  "why  it  happened"  questions.  The  next-generation 
system  should  include  artificial  intelligence  Components 
designed  to  aid  the  trainer  to  select  and  generate  a  limited 
number  of  aids  for  display  during  the  AAR.  In  addition  to 
"ground  truth"  aids,  this  system  should  also  display  aids  to 
address  the  "how  to  improve"  question.  Typically,  AAR  leaders 
identify  a  relevant  diagram  from  a  "How  To  Fight"  manual,  which 
must  then  be  displayed  separately  from  the  ATAFS  displays  (e.g., 
by  using  an  overhead  projector) .  The  next -generation  system 
should  also  include  a  library  of  such  slides  and  provide  the 
trainer  help  in  selecting  displays  and  inserting  them  in  the 
proper  points  in  the  AAR. 

Self-Assessment  Techniques 

In  examining  the  training  requirements  for  multiforce 
exercises,  Mirabella,  Siebold,  and  Love  (1998)  noted  that  the 
standard  approaches  for  measuring  AAR  performance  present  some 
distinct  disadvantages.  First,  the  typical  multiforce  exercise 
focuses  on  planning  and  executive  management  functions  rather 
than  on  the  tactical  control  and  performance  outcomes  that 
characterize  lower  echelons.  Thus,  automated  outcome  measures 
from  simulations  are  not  relevant  to  these  kinds  of  exercises. 
Second,  the  overhead  costs  for  0/Cs  in  a  complex  multiforce 
exercise  would  be  prohibitive.  Third,  many  of  the  processes 
pertaining  to  a  planning  exercise  are  covert  and,  therefore,  are 
not  subject  to  third-party  observation. 

Mirabella,  Siebold,  and  Love's  (1998)  solution  for 
providing  feedback  in  multiforce  exercises  is  to  base  the  AAR 
process  on  self-assessments  of  performance  that  are  moderated 
through  a  Delphi  process.  Abbreviated  or  "mini-"  AARs  are 
interspersed  throughout  these  extended  exercises .  Before  the 
mini-AAR,  participants  use  standardized  instruments  to  rate 
independently  the  performance  of  the  tactical  cell  to  which  they 
belong.  Then,  the  cell  leader  brings  the  individuals  together, 
reviews  all  ratings,  and  identifies  major  differences  in 
ratings.  The  cell  leader  then  conducts  a  discussion  among  cell 
members  to  promote  consensus .  This  procedure  results  in  a  set 
of  group  ratings  on  which  members  can  agree.  Note  that  the 
processes  of  individuals  independently  rating  performance, 
identifying  differences,  and  building  a  consensus  are  essential 
elements  of  the  Delphi  process  of  problem-solving. 
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Mirabella,  Siebold,  and  Love  (1998)  also  discuss  two 
enhancements  to  this  basic  procedure  that  can  be  used  to 
validate  self-assessments.  One  enhancement  specifies  that  those 
cells  receiving  outputs  from  the  cell  being  assessed  rate  the 
timeliness  and  quality  of  those  outputs.  The  other  enhancement 
is  to  have  0/Cs  complete  the  same  rating  instruments  that  cell 
members  complete  as  a  check  for  consistency  and  convergent 
validity.  Both  enhancements  are  designed  to  be  used  as 
resources  allow. 

Preliminary  applications  of  this  self-assessment  procedure 
have  been  encouraging.  However,  Mirabella  et  al .  conclude  that 
more  research  is  needed  to  validate  the  system  formally  for  use 
in  higher  echelon  AARs. 

Memory  and  Cognition 

Researchers  agree  that  the  AAR  is  not  simply  an  event  where 
exercise  participants  passively  receive  feedback;  rather,  it  is 
an  instructional  process  where  participants  learn  about  the 
causes  of  their  actions  and  develop  strategies  for  improving 
their  performance.  Because  the  AAR  is  a  learning  process, 
developers  have  recognized  the  relevance  of  memory  and 
cognition.  The  following  sections  discuss  topics  related  to  the 
learning  process  in  the  AAR. 

Transfer  of  Training 

In  the  present  context,  transfer  of  training  refers  to 
improvements  in  combat  performance  resulting  from  practice  on  a 
training  exercise.  One  of  the  most  fundamental  principles 
underlying  this  concept  is  that  the  amount  of  transfer  varies 
directly  with  the  similarity  of  conditions  between  training  and 
actual  combat.  In  accord  with  that  principle.  Army  doctrine 
encourages  leaders  to  "train  as  you  fight"  to  maximize  transfer 
of  training  from  simulations  to  combat. 

Transfer  of  training  principles  have  some  important 
implications  for  the  AAR.  One  is  that  the  discussion  should 
emphasize  the  conditions  of  combat,  not  those  of  the  simulation. 
Accordingly,  one  "rule  of  engagement"  for  conducting  the  AAR,  as 
stated  in  the  STAARS  Handbook ,  is  to  "...  discuss  tactics, 
not  gaming"  (TRADOC,  1997,  p.  13).  The  AAR  leader  must  also 
point  out  when  outcomes  from  a  simulation  do  not  correspond  to 
combat  outcomes  or  to  outcomes  from  other  simulations.  The 
intent  of  the  tactical  focus  is  to  increase  the  probability  that 
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competencies  acquired  in  the  AAR  will,  in  fact,  transfer  to 
combat  situations. 

Memory  Aiding 


The  AAR  learning  process  depends  heavily  on  memory. 

Although  technologies  such  as  TES,  virtual  simulation,  and 
automated  performance  measurement  may  help  substantiate  what 
happened,  participants'  memories  of  an  exercise  remain  an 
essential  source  of  information  for  understanding  what  events 
occurred  and  why  they  happened.  Further,  participants  depend  on 
their  memories  to  remember  the  suggested  changes  to  unit 
training  identified  during  the  AAR  so  that  they  can  implement 
these  changes  in  subsequent  performance  or  at  home  station. 

Several  aspects  of  the  AAR  are  specifically  designed  to  aid 
the  participants'  memories  for  these  questions.  The  AAR  session 
starts  with  a  review  of  "what  happened"  to  refresh  the 
participants'  memories  before  discussing  the  events.  The 
discussions  provide  additional  details  to  broaden  the 
understanding  of  events.  The  AAR  concludes  with  a  summary  to 
reinforce  the  AAR  points.  Finally,  the  THPs  provide  cues  for 
remembering  the  performance  events  and  training  solutions 
discussed  during  the  AAR.  As  such,  the  THP  is  designed  to  . 
provide  a  bridge  between  the  AAR  and  home  station  training. 

The  potential  effectiveness  of  these  AAR  memory  aids  is 
beyond  question.  The  question  is  whether  they  are,  in  fact, 
used  as  designed.  For  instance,  Fobes  and  Meliza  (1989)  found 
that  the  print-based  portions  of  the  THP  were  not  being  used  to 
develop  unit  training  plans  because : 

•  The  THPs  were  too  large  and  complex  (sometimes  over  1,000 
pages  in  length) . 

•  Some  of  the  detailed  information  was  unnecessary  and 
irrelevant,  and  important  details  (e.g. ,  nature  of 
deficiencies  and  personnel/unit  involved)  were  often 
omitted. 

•  There  were  inconsistencies  within  THPs  and  between  THPs 
and  AARs . 

•  Training  solutions  are  typically  not  provided  or  are 
buried  in  the  details. 
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It  would  be  interesting  to  find  out  if  present-day  THPs  suffer 
from  some  of  the  same  problems. 

Problem-Solving 

The  authors  of  A  Leader's  Guide  to  After-Action  Reviews 
(CAC,  1993)  explicitly  recognized  that  AAR  is  a  "problem-solving 
process"  (p.  4-3) .  As  has  been  described  previously,  exercise 
participants  are  faced  with  a  three-part  problem,  which 
corresponds  to  the  three  questions  that  comprise  the  focus  for 
the  AAR.  The  problem  posed  by  the  first  AAR  question  (What 
happened?)  is  relatively  simple  in  that  it  can  be  answered  by 
studying  the  recorded  portions  of  the  battle  and  recalling 
events  from  memory.  Answers  to  the  remaining  two  questions  (Why 
did  it  happen?  and  How  can  performance  be  sustained  or 
improved?)  are  clearly  more  challenging  since  they  require 
participants  to  make  inferences  about  the  causes  and  outcomes  of 
their  actions. 

Many  theories  describe  problem-solving  and  decision-making 
processes,  and  prescribe  procedures  and  training  techniques  for 
enhancing  these  interrelated  processes.  Most  of  these  theories 
fall  on  a  continuum — ^with  formal,  analytic  models  on  one  end  and 
naturalistic,  perception-based  models  on  the  other  end. 

According  to  the  formal  model  (e.g.,  Keeney,  1982),  decision¬ 
making  is  a  linear  process.  Participants  solve  a  problem  by 
breaking  it  down  into  a  linear  sequence  of  steps.  An  example 
series  of  steps  might  be  described  as  follows: 

•  Determine  goals  and  objectives. 

•  Develop  alternative  courses  of  action. 

•  Compare  and  evaluate  courses  of  action  on  criteria 
consistent  with  goals  and  objectives. 

•  Choose  course  that  best  matches  criteria. 

•  Implement  the  decision. 

The  naturalistic  model  of  decision-making  (e.g.,  Zsambok 
&  Klein,  1997)  assumes  that  most  real-world  decisions  are  not — 
and  cannot  be — made  in  such  a  slow  and  deliberate  fashion. 
Instead,  this  model  maintains  that  decisions  are  based  on  a 
recognition  process  that  matches  the  present  problem  with  one  or 
more  similar  problems  that  the  problem  solvers  remember  having 
faced  in  the  past.  Thus,  the  former  model  emphasizes  formal 
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analysis,  whereas  the  latter  model  stresses  experience  in  the 
problem-solving  process. 

Mental  Models 

A  mental  model  is  an  internal  construct  of  a  learner's 
physical  or  social  world  to  remember,  think  about,  and  explain 
events  and  concepts.  In  the  context  of  the  AAR,  mental  models 
help  explain  the  events  that  occurred  during  the  simulated 
exercise.  As  participants  acquire  more  diverse  details  about 
exercise  events,  their  mental  models  become  more  elaborate. 

More  elaborate  mental  models  are  easier  to  remember  and 
manipulate  and,  thus,  are  associated  with  enhanced  performance. 
Moreover,  because  AARs  typically  involve  teams  of  individuals 
(e.g.,  staffs,  crews,  platoons).,  they  should  share  identical 
(or,  at  least,  very  similar)  mental  models  so  that  they  have  a 
common  understanding  of  events  and  their  causes. 

A  particularly  effective  method  for  providing  more 
elaborate  details  about  the  simulated  battle  is  to  hear  and 
react  to  input  from  others.  The  AAR  embodies  this  idea  by 
encouraging  participants  to  provide  their  experiences  and  listen 
to  interpretations  from  others.  This  technique  is  particularly 
effective  in  fostering  shared  mental  models  of  task  performance. 
Another  approach  is  to  have  members  of  other  teams  also  discuss 
the  events  from  their  points  of  view.  OPFOR  members  provide  a 
valuable  input  because  they  offer  a  diametrically  opposed  point 
of  view  on  the  exercise. 


Group  Dynamics 

In  addition  to  being  a  learning  process,  the  AAR  is  a 
social  process  in  that  participants  come  together  to  make 
collective  decisions  about  their  performance.  As  collective 
process,  the  facilitating  and  constraining  effects  of  group 
dynamics  are  relevant  to  AAR  processes.  Some  of  the  better 
known  effects  of  group  dynamics  are  discussed  below. 

Social  Effects  on  Performance 


Research  indicates  that  the  presence  of  others  has 
significant  effects  on  performance.  As  described  below,  these 
effects  may  be  either  positive  or  negative. 

Social  Facilitation.  The  AAR  benefits  from  the  positive 
effects  of  social  facilitation — ^which  is  the  enhancement  of 
performance  caused  by  the  mere  presence  of  other  humans.  This 
phenomenon  is  widely  observed  and  is  usually  attributed  to 
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increased  motivation  or  arousal  (e.g.,  Zajonc,  1965).  However, 
the  presence  of  others  may  depress  performance  when  a  new  and/or 
difficult  task  is  being  performed  or  when  this  presence  causes 
distractions.  These  exceptions  do  not  seem  to  apply  to  the  AAR 
for  two  reasons : 

1.  Factors  such  as  evaluation  apprehension,  fatigue,  and 
exercise  complexity  may  increase  difficulty  of  the  AAR; 
however,  participating  in  AARs  is  normally  a  routine 
activity  and  is  not  particularly  difficult. 

2 .  AAR  leaders  are  trained  to  keep  the  discussion  focused 
on  the  key  points  of  the  AAR.  While  leaders  encourage 
participants  to  speak  freely,  they  request  that 
participants  speak  one  at  a  time  to  minimize 
distractions . 


Another  limitation  to  social  facilitation  is  that  the 
function  relating  number  of  participants  to  motivation  and/or 
performance  is  often  depicted  as  being  non-linear  in  form.  That 
is,  the  effect  is  positive  up  to  small  or  moderately  sized 
groups  but  decreases  and  perhaps  turns  negative  as  group  size 
increases  beyond  some  given  number.  This  has  been  related  to  an 
oft-cited  generalization,  sometimes  referred  to  as  the  Yerkes- 
Dodson  law,  that  describes  the  effects  of  arousal  on  performance 
as  an  inverted  U-shaped  function:  the  effects  are  increasing  up 
to  some  optimal  point  and  decreasing  beyond  that.  In  addition 
to  this  theoretical  constraint,  there  is  a  practical  constraint 
to  social  facilitation:  it  is  simply  more  difficult  to  control 
the  interpersonal  distractions  in  larger  crowds  than  in  smaller 
crowds.  For  both  reasons,  some  optimal  size  probably  exists  for 
an  AAR.  However,  the  optimum  is  likely  to  vary  with  other 
variables,  such  as  the  format  of  AAR  (formal  or  informal) ,  the 
echelon  (platoon,  company,  battalion,  or  higher) ,  and  the  type 
of  simulation  (live,  virtual,  or  constructive) . 

Social  Loafing.  In  contrast  to  social  facilitation,  the 
related  concept  of  social  loafing  has  a  negative  effect  on 
performance.  Social  loafing  is  the  effect  of  decreased 
individual  performance  observed  in  larger  groups,  where 
individuals  let  others  in  the  group  do  their  work  (Latane, 
Williams,  &  Harkins,  1979) .  The  phenomenon  of  social  loafing 
appears  to  occur  when  the  group  produces  a  single  product  and 
iri^ii'vi duals  are  not  held  accountable  for  their  own  performance. 
Social  loafing  is  minimized  in  AAR  sessions  because  leaders 
encourage  everyone  to  participate  actively  in  the  discussion. 

In  this  regard,  one  technique  that  AAR  leaders  use  is  to  address 
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questions  directly  to  those  participants  who  do  not  appear  to  be 
contributing  to  the  discussion.  In  general,  the  best  advice  to 
prevent  social  loafing  may  be  to  enforce  the  AAR  rule  that 
requires  everyone  to  participate  actively  in  the  process  (e.g., 
Scott,  1983,  1984;  Scott  &  Fobes,  1982;  Bosley,  Onoszko,  Knerr, 

&  Sulzen,  1979) . 

Group  Identification  and  Cohesiveness .  The  AAR 
participants  are  not  a  group  of  unrelated  individuals;  rather, 
they  are  members  of  a  unit  or  team.  In  general,  behavioral 
research  has  shown  that  group  performance  improves  to  the  extent 
that  members  identify  with  and  are  committed  to  the  group.  The 
document  entitled  A  Leader' s  Guide  to  After-Action  Reviews (  CAC, 
1993)  acknowledges  this  effect  when  it  states  that  one  purpose 
of  the  AAR  is  to  "  .  .  .  promote  bonding  and  esprit  ..."  among 

unit  members  (p.  1) .  Siebold  and  Kelly  (1988)  developed  a  short 
questionnaire  designed  to  assess  the  cohesiveness  of  combat 
platoons.  In  a  subsequent  ARI  study,  Keesling,  Ford,  O' Mara, 
McFann,  and  Holz  (1992)  examined  the  relationship  of  responses 
to.  this  questionnaire  and  platoon  performance  at  NTC.  They 
found  a  relatively  small  but  positive  correlation  between 
performance  and  a  component  of  the  instrument  related  to  the 
leader  characteristics — specifically,  his  perceived  competency 
and  how  well  he  trains  with  his  platoon  members. 

A  recent  review  of  research  has  shown  that  the  apparent 
mechanism  for  the  positive  effects  of  group  cohesiveness  is  the 
group's  commitment  to  the  task  (Mullen  &  Copper,  1994). 

Moreover,  the  relationship  is  bidirectional:  cohesiveness 
increases  task  performance  by  increasing  commitment  to  the  task 
and  the  act  of  performing  the  task  together  increases 
commitment,  thereby  increasing  cohesiveness.  The  first 
component  of  this  bidirectional  relationship  implies  that 
increasing  cohesiveness  could  promote  better  performance  in  the 
AAR  and  in  subsequent  exercises.  The  second  component  implies 
that  cohesiveness  is  a  result  rather  than  a  cause  of  good 
performance  in  exercises — the  interpretation  favored  by  Keesling 
et  al.  (1992)  . 

Regardless  of  the  direction  of  the  relationship,  increased 
group  cohesiveness  is  a  valued  outcome.  In  that  regard,  many 
theories  and  findings  in  social  psychology  suggest  that  the 
following  factors  increase  group  cohesiveness: 

•  Increasing  intergroup  interactions. 

•  Inducing  agreement  on  group  goals  using  consensus - 
building. 


47 


For 


•  Increasing  intergroup  competition. 

•  Reducing  intragroup  competition  and  discord. 

•  Emphasizing  group  success. 

•  Maintaining  a  pleasant,  positive  atmosphere. 

Many  factors  that  increase  cohesiveness  are  operative 
during  the  AAR  or  are  actively  promoted  by  the  AAR  leader, 
instance,  the  fact  that  the  unit  has  just  fought  against  a 
common  enemy  (intergroup  competition)  should  increase  unit 
cohesiveness.  Kaplan  and  Fallesen  (1986)  stressed  the 
importance  for  the  AAR  leader  to  maintain  a  positive  atmosphere 
in  which  participants  can  disagree  without  fear  of  retribution. 

A  Leader's  Guide  to  After-Action  Reviews  (CAC,  1993)  echoes  this 
idea,  recommending  that  leaders  should  "...  create  a  positive 
atmosphere  conducive  to  maximum  participation"  (p.  4-2) .  In 
addition,  the  leader  is  advised  not  to  allow  blame  ("finger¬ 
pointing")  to  focus  on  any  member  or  group  of  members.  This 
reduces  the  threat  of  intragroup  competition,  which  could  reduce 
cohesiveness . 

Another  AAR  strategy  is  also  designed  to  increase  group 
cohesiveness:  the  advice  to  AAR  leaders  that  they  facilitate 

t>ut  not  take  part  in  the  discussion  concerning  group  performance 
P^oblstns  and  solutions.  Staying  out  of  this  process  promotes 
unit  integrity.  It  also  encourages  the  unit  to  develop  their 
own  training  solutions,  thereby  increasing  group  commitment 
toward  any  solution  that  they  derive.  As  Scott  (1983,  1984) 
suggested,  having  the  unit  members  discover  and  solve  their  own 
problems  promotes  "ownership"  of  the  problem  and  the  solution, 
and  increases  the  likelihood  that  the  solutions  are  actually 
implemented.  An  additional  implication  is  that  the  process  of 
agreeing  on  these  common  goals  should  also  provide  feedback  to 
promote  more  group  cohesion. 

Communication  Theory  and  Techniques 

AAR  success  is  highly  dependent  on  the  quantity  and  quality 
of  communication  among  participants  and  communication  between 
participants  and  the  AAR  leader.  Thus,  implications  for  the  AAR 
have  been  drawn  from  the  field  of  communication  theory  and 
techniques . 

Descriptive  Communications 

The  term  "descriptive  communications"  refers  to  a  set  of 
techniques  that  are  regarded  as  being  highly  effective  and  are 
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used  in  several  different  fields,  such  as  management, 
psychiatry,  and  law.  For  the  AAR,  Kaplan  and  Fallesen  (1986) 
described  descriptive  communications  as  methods  that  replace 
abstractions  with  specific  statements.  In  so  doing,  these 
methods  discourage  judgmental  comments  in  favor  of  more  specific 
comments  about  behavior.  Using  the  concept  of  descriptive 
communications.  Downs,  Johnson,  and  Fallesen  (1987,  p.  B-3) 
derived  the  following  four  prescriptions  for  the  AAR  leader: 

1.  Be  specific — abstractions  should  be  avoided.  Statements 
such  as  "I  didn't  get  a  good  feel  for  ..."  or  "We 
didn't  have  our  timing  down"  should  be  replaced  with 
more  specific  statements. 

2.  Be  thorough — avoid  the  inclination  "to  make  a  long  story 
short . " 

3 .  Focus  on  behaviors . 

4.  Refer  to  goals  and  how  successfully  or  unsuccessfully 
they  were  met . 


Use  of  descriptive  communications  techniques  should  have 
two  effects  on  AAR  processes.  First,  these  techniques  should 
increase  AAR  learning  by  increasing  the  probability  of 
successful  communication  (i.e.,  being  understood  by  others). 
Second,  these  methods  should  minimize  overly  judgmental 
comments,  thereby  decreasing  the  resentment  that  could  result 
from  negative  feedback.  This  latter  affective  consequence 
increases  the  probability  that  participants  will  implement  the 
strategies  at  home  station. 

Questioning  Techniques 

The  various  sources  for  AAR  guidance  consistently  agree  on 
the  importance  of  the  AAR  leader's  using  "open-ended"  questions. 
One  of  the  initial  ARI  sources  (Bosley,  Onoszko,  Knerr,  & 

Sulzen,  1979)  identifies  asking  open-ended  questions  as  an 
essential  skill  that  AAR  leaders  must  learn.  The  term  "... 
refers  to  questions  that  are  structured  to  allow  for  many 
acceptable  answers.  .  .  .  Open-ended  questions  are  used  to 

stimulate  thinking  and  to  elicit  discussion"  (Cameron,  et  al . , 
1997,  pp.  35-36).  A  Leader's  Guide  to  After-Action  Reviews 
(CAC,  1993)  advises  the  following: 

[The  AAR  leader]  should  not  ask  yes  or  no  questions, 
but  encourage  participation  and  guide  discussion  by 
using  open-ended  and  leading  questions.  An  open-ended 
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question  has  no  specific  answer  and  allows  the  person 
answering  to  reply  based  on  what  was  significant  to 
him.  Open-ended  questions  are  also  much  less  likely 
to  put  him  on  the  defensive  .  .  .  (p.  4-3) . 

Because  open-ended  questions  purportedly  stimulate 
discussion  and  thought  about  battle  incidents,  they  seem 

well  suited  to  addressing  the  causes  of  particular 
performance  problems  (i.e.,  the  "why"  question).  However, 
recent  research  suggests  that  open-ended  questions  may  also  be 
helpful  for  the  initial  question  (the  "what  happened"  question) . 
This  research,  which  is  based  on  naturalistic  or  "real-life" 
conception  of  memory,  has  shown  that  open-ended  questions  can 
improve  the  recall  of  facts  and  incidents  (Koriat  &  Goldsmith, 
1996) .  The  facilitation  is  thought  to  be  caused  by  the 
learners'  increased  control  of  their  own  memory  processes. 

To  the  extent  that  open-ended  questions  are  abstract,  their 
use  appears  to  contradict  guidance  from  descriptive 
communication  that  specific  comments  are  better  than  abstract 
ones.  In  fact,  results  from  ARI  research  (Downs,  Johnson,  and 
Fallesen,  1987)  suggest  that,  during  the  AAR,  specific  questions 
tended  to  elicit  specific  comments,  whereas  abstract  questions 
elicited  abstract  comments.  Downs,  Johnson,  and  Fallesen  (1987) 
suggest  that  perhaps  the  best  advice  is  to  mix  abstract  and 
specific  comments.  The  issue  here  appears  to  be  one  of  timing: 
open-ended  questioning  may  be  good  to  initiate  a  discussion,  but 
specific  questions  and  comments  may  be  needed  later  to  focus  on 
particular  incidents  and  causes. 

Form  and  Content  of  Feedback 


Downs,  Johnson,  and  Fallesen  (1987)  used  message  content 
analysis  methods  to  determine  the  form  and  content  of  feedback 
provided  in  AARs .  They  examined  videotaped  AARs  of  six 
battalion- level  exercises  on  the  ARTBASS.  Transcripts  of  AAR 
sessions  identified  individual  utterances,  which  were  defined  as 
uninterrupted  verbal  strings.  Trained  analysts  then  classified 
each  utterance  with  respect  to  certain  characteristics, 
iricluding  those  derived  from  a  prescriptive  model  of  performance 
feedback.  According  to  their  prescriptive  model,  feedback 
should  provide  four  types  of  information: 

1.  Performance  vs.  personal  characteristics.  Feedback 
should  be  directed  toward  correctable  behaviors  rather 
than  toward  personal  characteristics. 
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2.  Rationale.  Feedback  should  provide  a  rationale  for 
participants'  performance  to  explain  why  they  did  what 
they  did. 

3.  Goals.  Feedback  information  that  references  task  goals 
or  objectives  should  be  provided  along  with  specific 
information  about  performance. 

4.  Corrective  actions.  Feedback  should  provide  strategies 
for  continuing  effective  behaviors  and  changing 
ineffective  ones. 

Results  indicated  that  most  comments  came  from  the 
battalion  commander,  the  ARTBASS  leader,  and  company  commanders. 
Fewer  comments  came  from  battalion  staff  officers  (intelligence, 
operations,  logistics,  and  fire  support  staff  officers)  and  from 
the  OPFOR.  The  content  of  most  messages  concerned  the  adequacy 
of  information  provided  during  the  battle  (e.g.,  failure  to 
request  or  provide  the  appropriate  vital  information)  and 
coordination  issues  (e.g.,  expending  resources  too  quickly  and 
failure  to  understand  battle  objectives) .  With  regard  to  their 
prescriptive  model,  the  analysts  deemed  that  adequate  feedback 
was  provided  in  three  of  the  four  categories.  The  exception  was 
the  relative  infrequency  of  requests  for  and  submissions  of 
information  on  the  rationales  of  performance.  The  most  serious 
problem  noted  by  the  authors  was  the  AAR  leaders'  failure  to  use 
questions  to  guide  the  AAR  process. 

Instructional  Science 

Instructional  science  refers  to  the  practical  science  and 
technology  related  to  the  design,  development,  implementation, 
and  evaluation  of  training.  Because  the  AAR  is  a  pedagogical 
process,  the  principles  of  instructional  science  should  be  used 
to  enhance  the  effectiveness  of  the  AAR.  A  variety  of 
instructional  science  principles  have  been  applied  to  the  AAR 
process,  some  of  which  are  discussed  below. 

Guided  Discovery  Learning 

The  concept  of  "discovery  learning"  is  generally  attributed 
to  Jerome  Bruner  as  a  component  of  his  learner-based  theory  of 
instruction.  According  to  Bruner  (1961) ,  learning  is  a 
rearrangement  and  transformation  of  existing  knowledge.  This 
implies  that  learning  is  a  personal  and  active  process.  The 
best  method  for  achieving  true  insight  is  for  learners 
themselves  to  "discover"  the  solution  to  problems  based  on  their 
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own  knowledge  and  experience  and  with  little  or  no  intervention 
from  an  instructor.  In  his  model  of  instruction,  the  teacher 
does  not  lecture;  rather,  the  teacher  facilitates  the  learning 
process.  The  AAR  process  is  consistent  with  this  model  since 
the  leader  facilitates  in  AAR  discussions  rather  than  directly- 
participating  in  these  discussions.  Through  the  process  of 
self-discovery,  moreover,  unit  members  achieve  a  better 
understanding  of  events,  causes  of  those  events,  and  solutions 
for  performance  problems. 

Early  in  the  development  of  the  AAR,  discovery  learning  was 
recognized  as  an  important  part  of  the  process  (e.g. ,  Shriver  et 
al.,  1975)  and  continues  to  be  acknowledged  in  more  recent 
publications  (e.g.,  Cameron  et  al . ,  1997;  Gubler,  1997;  TRADOC, 
1997) .  In  particular,  Gubler  (1997)  used  the  term  "guided 
discovery  learning"  to  imply  that  AAR  leaders,  although  they 
avoid  direct  participation  in  the  AAR  discussion,  provide 
direction  and  structure  to  the  process.  For  instance,  the  AAR 
leader  is  typically  advised  to  guide  the  discussion  and  to  keep 
it  focused  on  the  key  issues.  The  dilemma  is  this:  whereas 
some  guidance  may  be  needed  to  keep  the  AAR  on  track,  too  much 
guidance  could  turn  the  free-play  exercise  into  a  lecture. 
Research  should  be  addressed  to  determine  the  amount  and  kind  of 
guidance  that  is  actually  needed  in  the  AAR. 

Experiential  Learning 

Cameron  et  al .  (1997)  point  out  that,  in  addition  to 

discovery  learning,  the  AAR  is  based  on  experiential  learning 
concepts.  Experiential  learning  has  roots  in  the  "project 
method"  of  instruction  that  dates  to  the  early  part  of  this 
century.  This  approach  stipulates  that  learning  is  facilitated 
by  real-world  experiences  and  is  therefore  often  associated  with 
the  phrase  "learn  by  doing."  This  is  not  to  deny  the  role  of 
more  abstract  forms  of  learning.  In  fact,  starting  in  the  early 
1980s,  formal  theories  of  experiential  learning  have 
increasingly  specified  the  conditions  under  which  concrete 
experience  should  facilitate  more  abstract  forms  of  learning. 

For  instance,  Kolb  (1983)  has  described  a  sequential  and  cyclic 
model  of  learning  that  includes  the  four  stages  of  (1)  concrete 
experience,  (2)  reflective  observation,  (3)  abstract 
conceptualization,  and  (4)  active  experimentation.  Furthermore, 
active  experimentation  (stage  4)  restarts  this  cycle  by  causing 
the  apprehension  of  a  new  experience  (stage  1) .  Presumably,  the 
AAR  process  corresponds  to  stages  2  and  3,  whereas  the 
simulation  exercise  corresponds  to  stages  1  and  4. 
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A  key  feature  of  the  experiential  learning  approach  is  its 
emphasis  on  active  learning  through  experimentation.  In  that 
sense,  it  is  similar  to  discovery  learning.  There  are  two 
differences,  however.  First,  experiential  learning  is  more 
structured.  It  is  carefully  designed  by  teachers,  and  learners 
must  critically  reflect  on  their  experiences  (Fardouly,  1999) . 
Second,  discovery  learning  evolved  from  concept  learning 
research  on  individuals,  whereas  experiential  learning 
emphasizes  the  role  of  peer  group  interaction,  particularly  in 
providing  feedback  and  helping  others  to  reflect  on  the  results 
from  experience  (Cameron  et  al.,  1997). 

For  the  AAR  process,  the  implications  of  experiential 
learning  are  similar  to  those,  derived  from  discovery  learning: 
to  achieve  maximum  benefits,  unit  members  must  actively 
participate  in  the  process.  In  addition,  experiential  learning 
emphasizes  the  importance  of  feedback  from  others  who  share  the 
learning  experience.  Also,  the  cyclical  nature  of  experiential 
learning  implies  that  the  unit  realizes  the  full  benefit  from 
observation  and  conceptualization  provided  in  the  AAR  only  if  it 
follows  that  up  with  actively  experimenting  in  the  simulation 
and  experiencing  the  results.  In  many  ways,  the  iterative 
cycles  of  exercises  and  AARs  used  in  the  VTP  provide  an  ideal 
environment  for  experiential  learning. 

Cooperative  Learning 

Related  to  the  concept  of  social  facilitation  is  the 
instructional  concept  of  cooperative  learning.  This  concept 
stipulates  that  simply  providing  the  opportunity  for  students  to 
work  together  enhances  academic  achievement.  Although 
cooperative  learning  is  an  old  concept,  modern  research  on  the 
topic  dates  back  only  to  the  early  1970s.  Slavin  (1995) 
recently  reviewed  this  work  and  characterized  it  as  "  .  .  .  one 

of  the  greatest  success  stories  in  the  history  of  educational 
research"  (p.  1) .  Not  only  is  cooperative  learning  a  popular 
topic  for  educational  research  and  theory,  but  also  surveys  by 
Slavin  and  others  show  that  it  is  a  popular  teaching  method 
across  the  United  States. 

Slavin  (1995)  maintained  that  the  effects  of  cooperative 
learning  are  usually  explained  using  one  of  four  theoretical 
perspectives ; 

1.  Motivation.  Cooperative  learning  is  a  situation  in 

which  group  members  are  rewarded  for  group,  as  opposed 
to  individual,  performance.  This  situation  creates  an 
interpersonal  reward  structure  wherein  members  give  or 
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withhold  social  reinforcers  in  response  to  groupmates' 
task-related  efforts. 

2.  Social  cohesion.  Motivation  is  mediated  by  the 
cohesiveness  of  the  group.  That  is,  students  help  one 
another  learn  because  they  care  about  one  another  and 
want  one  another  to  succeed. 

3.  Developmental.  Interaction  among  students  around 
appropriate  tasks  increases  their  mastery  of  critical 
concepts . 

4.  Cognitive.  Peer  tutoring  increases  the  cognitive 
restructuring,  or  elaboration,  of  to-be-learned 
material.  The  facilitation  helps  both  tutor  and  tutee — 
but  especially  the  former. 

Slavin' s  review  provided  evidence  suggesting  that  all  four 
theoretical  perspectives  may  influence  achievement.  However, 
the  evidence  for  the  motivational  interpretation  was  most 
convincing,  because  almost  any  cooperative  learning  manipulation 
could  be  enhanced  by  the  addition  of  group  goals  and  individual 
accountability.  The  exception  is  instruction  on  tasks  that  are 
high  in  cognitive  complexity — that  is,  tasks  without  a  well- 
defined  path  to  the  solution  or  a  single  solution.  In  this 
case,  it  appears  that  participating  in  discussions  and  debates 
even  listening  to  others  do  so  may  be  sufficient  to  enhance 
learning.  This  definition  of  a  cognitively  complex  task  fits 
the  task  that  the  participants  face  in  an  AAR  session. 

Like  experiential  learning,  the  research  on  cooperative 
learning  suggests  that  AAR  performance  may  be  enhanced  by  active 
interaction  among  participants.  The  research  on  cooperative 
learning  also  suggests  that  this  interaction  may  take  the  form 
of  group  discussions  and  even  arguments  among  participants. 
Further,  research  suggests  that  the  AAR  leader  does  not  have  to 
establish  a  complex  system  of  rewards  to  reap  the  benefits  of 
cooperative  learning. 


Systems  Concepts 

Lockheed  Martin  Federal  Systems  and  Illusions,  Inc.  (1996) , 
in  their  review  of  the  AAR  concept,  made  the  point  that  the  AAR 
was  originally  conceived  as  a  discrete  event  occurring  at  ENDEX, 
where  a  leader  mediates  and  facilitates  discussion.  This 
concept  was  especially  true  in  the  early  days,  given  that  most 
AARs  were  informal  in  nature.  However,  with  the  advent  of 
advanced  instrumentation  and  data  processing,  much  more 
information  can  now  be  integrated  into  the  AAR.  With  more 
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information  that  directly  supports  the  unit's  training 
objectives  and  increased  preparation  of  structured  materials, 
the  AAR  is  now  conceived  as  a  process .  This  process  view  of  the 
AAR  has  encouraged  the  application  of  systems  concepts  to  the 
design,  development,  and  delivery  of  the  AAR.  This 
system-analytic  interpretation  of  AAR  processes  is  consistent 
with  the  Army's  Systems  Approach  to  Training  (SAT),  which  is  a 
more  general  model  for  the  design,  development,  and 
implementation  of  training. 

Analysis  of  Processes 

One  of  the  distinguishing  features  of  systems  concepts  of 
instruction  is  the  analysis  of  training  development  and 
implementation  into  sequential  phases.  For  instance,  the 
original  Interservice  Procedures  for  Instructional  Systems 
Development  (IPISD)  divided  training  development  into  five 
phases:  analyze,  design,  develop,  implement,  and  control 

(Branson  et  al . ,  1975).  In  turn,  each  of  these  phases  was 
broken  down  into  smaller  units  of  activities  called  "blocks," 
which  were  broken  into  individual  steps. 

In  similar  fashion,  Kaplan  and  Fallesen  (1986)  described 
the  AAR  process  for  ARTBASS  exercises  as  comprising  four  phases: 
determine  training  objectives,  observe  the  exercise,  prepare  the 
AAR,  and  conduct  the  AAR.  Each  phase  was  analyzed  further  into 
multiple  sequential  steps.  Lockheed  Martin  Federal  Systems  and 
Illusions,  Inc.  (1996)  portrayed  the  AAR  process  as  comprising 
only  three  phases:  preparation,  execution,  and  review.  The 
latter  phase  encompassed  both  the  preparation  and  conduct  of  the 
AAR.  These  two  analyses  provide  similar,  but  not  identical, 
views  of  the  AAR  process . 

Evaluation  Component 


An  essential  element  of  any  systems  approach  to 
instructional  development  is  the  assessment  or  evaluation 
process,  which  functions  as  feedback  to  the  system.  For 
instance,  the  control  phase  of  IPISD  fulfills  just  this 
function.  In  contrast,  the  process  models  of  the  AAR  (Kaplan  & 
Fallesen,  1986;  Lockheed  Martin  Federal  Systems  and  Illusions, 
Inc.,  1996)  lack  this  crucial  process.  We  need  an  evaluation 
procedure  specifically  designed  for  the  AAR  process.  Such  a 
procedure  would  allow  participants  and  third-party  observers  to 
evaluate  AAR  sessions  by  established  principles  and  practices. 
Crissey  (publication  in  preparation)  identifies  some  AAR 
objective  performance  measures  that  could  be  included:  ratio  of 
unit  to  AAR  leader  comments,  number  of  solutions 
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developed/planned,  whether  or  not  problem  solution  is  reinforced 
through  retraining,  and  time  between  training  and  the  AAR  and 
between  the  AAR  and  retraining.  The  instrument  should  also 
include  subjective  but  structured  items  on  the  perceived 
effectiveness  and  efficiency  of  the  AAR  process.  This  proposed 
instrument  would  provide  feedback  to  AAR  leaders,  who  would  use 
the  information  to  improve  their  performance,  and  it  would 
provide  data  for  research  on  the  AAR  process. 

Embedded  System 

Another  hallmark  of  systems  models  is  embeddedness:  small 
systems  are  often  embedded  in  larger,  more  encompassing  ones. 
Accordingly,  the  method  for  developing  structured  training 
(C.  H.  Campbell  et  al.,  1997)  depicts  the  AAR  process  as 
subsystems  within  a  larger  system  describing  development  and 
delivery  of  structured  simulation-based  training.  In  this 
model,  the  AAR  process  is  divided  into  separate  development  and 
implementation  subprocesses,  which  are  embedded  into  two  of  the 
four  components  of  the  overall  model  as  described  below: 

1.  Needs  analysis.  Determine  the  critical  tasks  and  the 
resources  needed  to  train  them. 

2.  Development  of  training  support  package.  Develop  all 
materials,  including  scenario  and  tactical  materials, 
guides  for  all  support  personnel,  and  training 
materials,  which  contain  observation  guides  and  AAR 
materials. 

3.  Training  processes.  Identify  and  provide  cues  from  the 
simulation,  scripted  messages,  and  role  players. 

4 .  Learning  processes .  Monitor  performance  and  feedback  on 
critical  tasks,  and  execute  the  AAR. 

This  embedded  systems  model  makes  the  point  that  AARs  do  not 
exist  in  isolation.  AARs  have  implications  for  the  development  and 
evaluation  of  simulation  exercises,  and  the  proposed  systems  model 
of  AAR  processes  must  include  inputs  from  and  outputs  to  larger, 
more  inclusive  instructional  systems. 
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Research  Opportunities 

Our  analysis  of  the  principles  that  underlie  the  AAR 
suggested  several  potential  areas  for  research.  Some  of  these 
areas  are  described  below. 

The  After  Action  Review  (AAR)  and  Other  Sources  of  Feedback 

As  suggested  by  Brown,  Nordyke,  et  al.  (1998),  the  AAR 
should  be  viewed  in  the  context,  of  all  sources  of  feedback. 
Current  simulation-based  training  should  be  examined  to  identify 
appropriate  and  inappropriate  sources  of  feedback,  including 
that  provided  in  an  after-exercise  session  facilitated  by  a 
leader.  Hoffman  et  al .  (1995)  took  a  larger  perspective  in 

their  analysis  of  SIMUTA  training  materials,  but  their  analysis 
and  conclusions  focused  narrowly  on  the  methods  and  simulations 
used  in  the  RCVTP. 

We  propose  that  the  requirements  for  feedback  be  examined 
for  the  general  case .  The  proposed  method  would  be  to  review 
the  literature  to  answer  basic  questions  about  when  and  where 
feedback  should  be  presented.  For  instance,  this  research  could 
determine  whether  it  would  be  better  to  present  certain  t^es  of 
feedback  during  the  exercise  (as  coaching  and  mentoring)  or 
after  the  exercise.  By  viewing  the  AAR  in  a  larger  context,  we 
see  how  it  fits  in  with  other  complementary  feedback  systems. 

Compliance  with  After  Action  Review  (AAR)  Principles 

The  principles  that  underlie  the  AAR  are  relatively  well 
known.  They  are  presented  in  Army  documents  (e.g.,  TC  25-20) 
and  are  discussed  in  detail  in  the  research  reports  cited  in 
this  document.  However,  it  would  be  useful  to  document  the 
extent  to  which  the  principles  outlined  in  this  report  and  other 
Army  documents  are  complied  with  in  practice.  Gubler  (1997) 
attempted  to  do  this,  but  his  attempt  was  based  on  only 
seventeen  videotaped  platoon  and  company  AARs  from  the  NTC  and 
JRTC.  This  sort  of  work  should  be  expanded  to  include  examples 
of  AARs  from  higher  echelons  and  from  other  training  sites,  such 
as  ARTEPs  conducted  at  home  station  and  collective  exercises 
performed  at  simulation  centers. 

A  model  of  effective  AARs  should  be  synthesized  from  the 
research  literature  and  military  guidance.  The  model  would 
suggest  appropriate  methods  for  research.  For  instance,  one 
part  of  the  model  might  relate  to  methods  for  enhancing  memory 
for  feedback.  In  that  case,  it  would  be  useful  to  track, 
through  interview  or  systematic  observation,  the  extent  to  which 
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AAR  practices  enhance  memory.  For  example,  do  participants 
engage  in  active  discussions  that  elaborate  on  memory  content? 
When  such  discussions  occur,  do  all  participants  remember  or 
understand  the  events?  For  AAR  summaries,  one  could  similarly 
ask  the  following  questions:  Do  AAR  leaders  provide  a  summary? 
Is  the  content  of  AAR  summaries  sufficient  to  aid  memory  and 
understanding?  Do  participants  remember  these  summaries? 

The  model  of  effective  AARs  may  provide  a  potentially 
useful  by-product  of  this  research  because  it  could  be  used  to 
derive  a  checklist  of  activities  that  should  occur  during  an 
AAR.  This  checklist  could  serve  as  an  instrument  for  evaluating 
AARs.  Ironically,  AAR  leaders,  who  provide  feedback  to  units, 
do  not  have  a  standardized  instrument  for  receiving  feedback  on 
their  own  performance. 

Utility  of  Take-Home  Packages  (THPs)  and 
Related  After  Action  Review  (AAR)  Products 

This  issue  was  last  addressed  when  Fobes  and  Meliza  (1989) 
conducted  interviews  in  March  and  April  of  1987  on  the  utility 
of  THPs  from  NTC.  As  documented  in  this  present  study,  much  has 
been  learned  and  developed  in  the  10  years  since  their  research. 
It  would  be  useful  to  compare  and  contrast  THPs  from  NTC  with 
the  automated  THPs  that  are  used  at  the  VTP.  It  would  also  be 
useful  to  revisit  this  issue  in  a  wider  context  than  NTC, 
including  various  instances  of  live,  virtual,  and  constructive 
simulations . 


Problem-Solving  and  Decision-Making 

The  consensus  among  researchers  and  military  writers  is 
that  AAR  sessions  involve  problem-solving  and  decision-making 
processes.  However,  writers  have  not  specified  the  nature  of 
these  internal  processes  as  they  apply  to  the  AAR.  To  enhance 
these  key  processes,  a  more  detailed  concept  of  these  processes 
has  to  be  developed. 

One  research  approach  might  be  to  compare  and  contrast  the 
processes  that  are  characteristic  of  successful  and  unsuccessful 
AAR  sessions.  Findings  may  help  determine  whether  decision¬ 
making  follows  the  formal  model  (e.g.,  Keeney,  1982)  or  a 
recognition-based  process  (e.g.,  Zsambok  &  Klein,  1997). 

Findings  from  such  exploratory  research  might  lead  to  subsequent 
research  on  appropriate  prompting  and/or  aids  for  AAR  sessions. 
If  the  formal  model  of  decision-making  were  demonstrated  to  be 
applicable  to  the  AAR,  the  literature  on  applied  decision-making 
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could  be  used  to  improve  the  AAR  process.  If,  on  the  other 
hand,  the  research  confirmed  that  successful  AAR  sessions  are 
characterized  by  a  recognition-based  process  as  opposed  to  an 
analytic  one,  the  instructional  approach  may  need  to  center  on 
drawing  from  or  building  up  the  participants'  base  of 
experience.  For  instance,  AAR  leaders  may  want  to  construct 
prompts  to  get  participants  to  use  their  experience  in  their 
discussions.  Alternatively,  if  the  experience  base  of  the 
participants  were  limited,  the  leader  could  supply  appropriate 
cases  as  appropriate.  Clearly,  both  models  have  implications 
for  the  conduct  of  AARs . 

Group  Cohesiveness  Measures 

The  research  literature  implies  that  group  cohesiveness  can 
have  two  relations  with  AAR  effectiveness.  Cohesiveness  can 
contribute  to  AAR  effectiveness,  and  it  can  also  be  an  outcome 
of  the  AAR.  Empirical  research  should  be  performed  to  determine 
whether  one  or  both  of  these  implications  are  true. 

ARI  has  developed  a  questionnaire  for  assessing  the 
cohesiveness  of  combat  platoons  (Siebold  &  Kelly,  1988) .  The 
methods  and  items  used  to  develop  this  instrument  could  be  used 
to  assess  the  cohesiveness  of  higher  echelons.  To  differentiate 
causes  from  effects,  experiments  could  be  devised  that  vary  the 
time  of  administration  and/or  conditions  that  affect 
cohesiveness . 

Depending  on  the  findings  of  the  research,  there  should  be 
important  implications  for  cohesiveness  and  the  AAR  process.  If 
cohesiveness  is  shown  to  be  a  positive  factor  in  the  AAR 
process,  the  AAR  designers  and  leaders  should  actively  promote 
the  development  of  group  cohesiveness  before  and  during  the  AAR 
session.  If  cohesiveness  is  an  outcome  from  an  effective  AAR, 
measures  of  cohesiveness  should  be  incorporated  as  an  important 
index  of  AAR  effectiveness. 

Computer-Aided  Analysis  of  After  Action  Review  (AAR)  Content 

Downs,  Johnson,  and  Fallesen  (1987)  provided  much  insight 
into  the  nature  of  AAR  feedback  through  detailed  content 
analysis  of  videotaped  AARs.  However,  the  number  of  AAR  samples 
was  greatly  limited  because  of  the  labor-intensive  nature  of 
content  analysis.  Since  that  time,  software  has  been  created  to 
speed  the  content  analysis  process.  This  improved  technology 
could  greatly  increase  the  potential  number, and  range  of  AARs 
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included  in  a  sample  and  reduce  concerns  about  reliability  of 
measurement . 
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Appendix  A:  Acronym  List 

AAR  .  After  Action  Review 

ACR  .  Advanced  Concepts  and  Requirements 

ARI . U.S.  Army  Research  Institute  for  the  Behavioral 

and  Social  Sciences 
ARNG . Army  National  Guard 

ARTBASS  . . Army  Training  Battle  Simulation  System 

ARTEP  .  Army  Training  and  Evaluation  Program 

ATAFS  .  Automated  Training  Analysis  and  Feedback  System 

BDA  .  Battle  Damage  Assessment 

BDS-D  .  Battlefield  Distributed  Simulation-Developmental 

C"*!  . Command,  Control,  Communications,  Computers,  and 

Intelligence 

CAC . Combined  Arms  Center 

CCTT .  Close  Combat  Tactical  Trainer 

CGF .  Computer-generated  Forces 

CITT  .  Commander's  Integrated  Training  Tool 

COBRAS  .  Combined  Arms  Operations  at  Brigade  Level 

Realistically  Achieved  Through  Simulation 
COS . Change  of  Status 

CTAFS  .  C^l  Training  Analysis  and  Feedback  System 

CTC  .  Combat  Training  Center 

DARPA  .  Defense  Advanced  Research  Projects  Agency 

DIS  .  Distributed  Interactive  Simulation 

DOS . Disk  Operating  System 

ENDEX . End  of  the  Exercise 

FY . Fiscal  Year 

IPISD  .  Interservice  Procedures  for  Instructional  Systems 

Development 

JCOS  .  Joint  Countermine  Operational  Simulation 

JRTC  .  Joint  Readiness  Training  Center 

JSIMS  .  Joint  Simulation  System 

JWFC  .  Joint  Warfighting  Center 

KR . .  Knowledge  of  Results 

MDT2  .  Multi -Service  Distributed  Training  Testbed 

METT-T  .  Mission,  Enemy,  Terrain,  Time,  and  Troops 

MILES  .  Multiple  Integrated  Laser  Engagement  System 

ModSAF  .  Modular  Semi -automated  Force 

MTP  .  Mission  Training  Plan 

NSC  .  National  Simulation  Center 

NTC  .  National  Training  Center 

0/C  .  Observer/Controller 

OPFOR . Opposing  Force 

ORD  .  Operational  Requirements  Document 

PDU . Protocol  Data  Unit 


A-1 


prime  .  Precision  Range  and  Integrated  Maneuver  Exercise 

R&D  .  Research  and  Development 

RCVTP  .  Reserve  Component  Virtual  Training  Program 

RPA  .  Research,  Development,  and  Acquisition 

REALTRAIN  .  Realistic  Training 

RSTA  .  . . Reconnaissance,  Surveillance,  and  Target 

Acquisition 

SAPOR . Semi -Automated  Force 

SAT  .  Systems  Approach  To  Training 

SATS  .  Standard  Army  Training  System 

SCOPES  .  Squad  Combat  Operations  Exercises  (Simulation) 

SGT  .  Staff  Group  Trainer 

SIMBART  .  Simulation-Based  Mounted  Brigade  Training 

SIMNET  .  Simulation  Networking 

SIMUTA  .  Simulation-Based  Multiechelon  Training  for 

Armor  Units 

SIMUTA-D  .  Simulation  Unit  Training  Assembly-Digital 

SOP  .  Standard  Operating  Procedure 

STAARS  . . Standard  Army  After  Action  Review  System 

STRICOM  .  Simulation,  Training,  and  Instrumentation  Command 

T&E . Test  and  Evaluation 

TADSS  .  Training  Aids,  Devices,  Simulations,  and 

Simulators 

TAFF-Aids  .  Training  Analysis  and  Feedback  Aids 

TC  .  Training  Circular 

TSMO . .  Training  Exercises  and  Military  Operations 

TES  .  Tactical  Engagement  Simulation 

THP  .  Take-Home  Package 

TRADOC  .  U.S.  Army  Training  and  Doctrine  Command 

TSP  .  Training  Support  Package 

UPAS  .  Unit  Performance  Assessment  System 

VTP  .  Virtual  Training  Program 

WF  XXI . .  Warfighter  XXI 
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